Skip to content

Provision InferenceClusters on CoreWeave - #655

Merged
negz merged 5 commits into
modelplaneai:mainfrom
negz:weave-it-to-me
Nov 7, 2026
Merged

negz merged 5 commits into
modelplaneai:mainfrom
negz:weave-it-to-me

Conversation

@negz

@negz negz commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Description of your changes

Fixes #94.

Modelplane can't provision clusters on CoreWeave. This PR adds CoreWeave as an InferenceCluster source, backed by a new CoreWeaveCluster XR built on provider-coreweave.

apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: coreweave-us-east-04
spec:
  cluster:
    source: CoreWeave
    coreweave:
      zone: US-EAST-04A
  nodePools:
  - name: gpu-rtxp6000
    className: rtxp6000-8x   # provisioning.coreweave.instanceType: rtxp6000-8x
    nodeCount: 1

CKS has no cloud API for node pools. A pool is a NodePool resource inside the cluster, so compose-coreweave-cluster composes each pool as a provider-kubernetes Object once the cluster is running. Usages block deleting the cluster and its kubeconfig until those Objects finish deleting. CKS's autoscaler sets an autoscaled pool's targetNodes, so that pool's Object only creates and deletes it. A second Object, under its own field manager, applies everything but targetNodes. Its apply leaves targetNodes in place and updates the pool's bounds, labels and taints.

CKS doesn't publish a kubeconfig. Its API endpoint accepts the same CoreWeave API token the provider uses. The function renders a kubeconfig from the cluster's endpoint and the token in the CoreWeave ProviderConfig's Secret. That puts a copy of the org's token in a Secret beside the CoreWeaveCluster, which is unavoidable until provider-kubernetes and provider-helm can read the token from provider-coreweave's Secret (crossplane-contrib/provider-kubernetes#603).

CoreWeave manages the GPU driver, GPU Operator and NFD, so the serving stack's CoreWeave half installs cert-manager, the NVIDIA DRA driver and a DCGM exporter, and GPU pools get the label that keeps the managed device plugin off them. CoreWeave runs its own DCGM exporter too, but it only attributes GPUs the device plugin allocated. The collector skips it. CKS's own RWX class, shared-vast, retains a volume after its claim goes, so ModelCache uses a composed copy with reclaimPolicy: Delete.

CKS gives LoadBalancer Services private addresses unless annotated for a public one. A cloud half of the serving stack can now declare the Service settings its load balancers need, which go into the GatewayClass's EnvoyProxy. The InferenceGateway's EnvoyProxy sets mergeType: StrategicMerge to inherit them, so neither gateway function knows which cloud it's on.

A Usage blocks deleting the VPC until the cluster finishes deleting, because deleting a VPC a cluster still uses can leak it (upbound/provider-upjet-coreweave#4). Like the AKS and EKS autoscalers, CKS's can't scale a pool up from zero nodes when its pods' GPUs bind through DRA (#345). The XRD's minNodeCount description says to keep it at 1 or higher.

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes.
  • Added or updated tests covering any composition function changes.
  • Signed off every commit with git commit -s.

@negz

negz commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown

Docs preview: https://modelplane-docs-pr-655.vercel.app (ready once the site's Content workflow finishes)

@modelplane-review modelplane-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this needs another round before it merges. The gateway annotation and the copied API token are design questions to settle first.

Most of the change follows the other provisioned clouds, and the questions are where CKS differs. Nothing has run on a real CKS cluster yet. One run would settle the GPU assumptions the description flags, and show whether the readiness and autoscaling problems raised inline happen in practice.

Reviewed by modelplane-review d148f8c. Reply, or react 👍 or 👎, on each comment to help tune it.

Comment thread docs/content/platform/providers.md Resolved
@negz

negz commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

/review

@modelplane-review modelplane-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes since my last review look close to ready, with a few questions left. The main one is whether the system pool's new readiness check can mark a serving cluster un-Ready when CKS replaces its one node.

The VPC Usage, the compose-usages step and the reworked gateway Service settings raise nothing new. The other findings are docs and code comments the changes made untrue.

The docs commit's message still says ModelCache uses shared-vast, and the CoreWeaveCluster commit says every pool starts with at least one node, where only autoscaled pools do.

Reviewed by modelplane-review 84d2a55. Reply, or react 👍 or 👎, on each comment to help tune it.

@negz

negz commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

/review

@modelplane-review modelplane-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes since my last review look ready to merge, apart from two wording points inline.

The system pool's readiness switch and deleting the StorageClass with the XR answer the earlier threads, and the new test cases cover both paths the switch can take.

Reviewed by modelplane-review 3017043. Reply, or react 👍 or 👎, on each comment to help tune it.

@negz

negz commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

I've run this end to end on a real CoreWeave org in US-EAST-04A. An InferenceCluster provisioned a VPC, a CKS cluster and its NodePools. The serving stack installed, both gateways got public IPs, and a chat completion went through the InferenceGateway to a vLLM replica on an RTX PRO 6000 node, using a GPU allocated through DRA. Foreground deletion tore it all down in about 7 minutes and left nothing in CoreWeave. The org only has quota for rtxp6000-8x, so that run used a local commit that puts the system pool on rtxp6000-8x instead of a CPU instance type.

On the live cluster:

  • Modelplane's DRA driver runs alongside the one CoreWeave installs for ComputeDomains, and publishes a gpu.nvidia.com ResourceSlice for every GPU.
  • CoreWeave's managed GPU Operator honors nvidia.com/gpu.deploy.device-plugin=false, which keeps the device plugin off GPU pools.
  • A CoreWeave mutating webhook sets externalTrafficPolicy: Local on every LoadBalancer Service, overriding Modelplane's Cluster pin. Both gateways still work.
  • CKS rejects a NodePool without targetNodes. An apply that omits it leaves it in place while another field manager owns it.
  • shared-vast keeps a volume after its claim goes. The copy with reclaimPolicy: Delete provisions and deletes volumes as expected.
  • CoreWeave's API answers a request to delete a VPC that a cluster still uses with NotFound, which provider-coreweave reads as already deleted.

Only unit tests cover what changed after that run: the system pool's Object waiting for a node only until the cluster is first Ready, and autoscaled pools' second Object. I'll run them live when I next have the org.

haarchri added a commit to haarchri/modelplane that referenced this pull request Nov 6, 2026
UpCloud's load balancer needs - TCP frontend mode and names that survive
the cloud controller's 64-character truncation - were plumbed as
serviceAnnotations through the InferenceCluster status and ServingStack
spec, with a {gateway} template and a per-Gateway EnvoyProxy that
restated the externalTrafficPolicy pin. The annotation also broke an
InferenceGateway with spec.tls: its Service has two ports, and the
controller matches config frontends to ports by name, so the single
unnamed frontend left the load balancer one nameless frontend on port 0
with no backend.

Instead, a cloud half now declares what its load balancers need on every
gateway Service as a GATEWAY_SERVICE subset of Envoy Gateway's
KubernetesServiceSpec, which join() folds into the GatewayClass-level
EnvoyProxy. The inference gateway's EnvoyProxy merges onto it with
mergeType: StrategicMerge, and on UpCloud alone overrides the config
annotation with a named TCP frontend per listener and names its Service.
The controller already prefixes every load balancer with the cluster
UUID, so a Service name that differs from the cluster gateway's within
the truncated remainder is enough; the UUID-built name, the {gateway}
template and UpCloudCluster's clusterId status all go. This is the
mechanism modelplaneai#655 introduces for CoreWeave, applied ahead of it so both
clouds share one shape.

The autoscaler fork lists every node group in the cluster each refresh
and gives any group missing from its --nodes list a minimum of one node
and a maximum of the cluster plan's limit, so it could scale the system
group down to one node or a fixed GPU pool up on pending pods, fighting
Crossplane as it reset each back to its nodeCount. Whenever any pool
autoscales, the group list now also pins the system group and every
fixed pool at their node count. compose_autoscaler_secret takes the
Autoscaler compose() already narrowed, a test now covers the node-group
sentinel that keeps groups composed through a Ready regression on
clusters without an autoscaler, and the node pool docs stop suggesting a
System pool beside the injected one, note that Premium and Starter plans
ignore diskSizeGb, and drop the stale GPU-pools-only wording.

Works around UpCloudLtd/upcloud-cloud-controller-manager#14.

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
haarchri added a commit to haarchri/modelplane that referenced this pull request Nov 6, 2026
UpCloud's load balancer needs - TCP frontend mode and names that survive
the cloud controller's 64-character truncation - were plumbed as
serviceAnnotations through the InferenceCluster status and ServingStack
spec, with a {gateway} template and a per-Gateway EnvoyProxy that
restated the externalTrafficPolicy pin. The annotation also broke an
InferenceGateway with spec.tls: its Service has two ports, and the
controller matches config frontends to ports by name, so the single
unnamed frontend left the load balancer one nameless frontend on port 0
with no backend.

Instead, a cloud half now declares what its load balancers need on every
gateway Service as a GATEWAY_SERVICE subset of Envoy Gateway's
KubernetesServiceSpec, which join() folds into the GatewayClass-level
EnvoyProxy. The inference gateway's EnvoyProxy merges onto it with
mergeType: StrategicMerge, and on UpCloud alone overrides the config
annotation with a named TCP frontend per listener and names its Service.
The controller already prefixes every load balancer with the cluster
UUID, so a Service name that differs from the cluster gateway's within
the truncated remainder is enough; the UUID-built name, the {gateway}
template and UpCloudCluster's clusterId status all go. This is the
mechanism modelplaneai#655 introduces for CoreWeave, applied ahead of it so both
clouds share one shape.

The autoscaler fork lists every node group in the cluster each refresh
and gives any group missing from its --nodes list a minimum of one node
and a maximum of the cluster plan's limit, so it could scale the system
group down to one node or a fixed GPU pool up on pending pods, fighting
Crossplane as it reset each back to its nodeCount. Whenever any pool
autoscales, the group list now also pins the system group and every
fixed pool at their node count. compose_autoscaler_secret takes the
Autoscaler compose() already narrowed, a test now covers the node-group
sentinel that keeps groups composed through a Ready regression on
clusters without an autoscaler, and the node pool docs stop suggesting a
System pool beside the injected one, note that Premium and Starter plans
ignore diskSizeGb, and drop the stale GPU-pools-only wording.

Works around UpCloudLtd/upcloud-cloud-controller-manager#14.

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
@negz

negz commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

I ran what changed since my last comment on a fresh cluster in US-EAST-04A:

  • The system pool's Object waited for a node, then switched to the Validated query once the CoreWeaveCluster was Ready.
  • Adding maxNodeCount to a fixed pool made it autoscale. The second Object applied autoscaling, minNodes and maxNodes, and targetNodes stayed with the first.
  • Lowering minNodeCount on the autoscaled pool updated the NodePool's minNodes.
  • With the model deleted, the autoscaler took the idle pool to zero and became the owner of targetNodes. Neither Object reverted it, and the cluster was Ready throughout.
  • Removing maxNodeCount made the pool fixed again. minNodes and maxNodes stay on the NodePool under the deleted second Object's field manager, but CKS ignores them while autoscaling is off, and accepts a targetNodes above the old maximum.
  • The serving stack's DCGM exporter ran beside CoreWeave's on the same node, and both kept publishing every profiling metric. Ours labelled the vLLM replica's GPU series with its deployment, engine, replica and role. CoreWeave's attributed a DRA-allocated GPU to its own HPC verification pod instead.

@negz

negz commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

/review

@modelplane-review modelplane-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes since my last review look close to ready, with a few questions on the second NodePool Object and the new DCGM exporter.

Splitting an autoscaled pool across two Objects looks sound, and your live run covers it. I'm unsure a fixed pool switched to autoscaling at zero nodes can ever scale up. I've also asked where the CKS-specific collector rule belongs, and flagged a docs page the exporter makes untrue.

The "Provision InferenceClusters on CoreWeave" commit message still says the CoreWeave half installs only cert-manager and the DRA driver, and doesn't mention the collector change.

Reviewed by modelplane-review 3017043. Reply, or react 👍 or 👎, on each comment to help tune it.

Signed-off-by: Nic Cope <nicc@rk0n.org>
@negz

negz commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

/review

@modelplane-review modelplane-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes since my last review look close to ready, with one question on whether an autoscaled pool's seed node still keeps it off zero.

The released provider pin and the updated telemetry and collector docs answer the earlier threads. Since a CKS pool that reaches zero stays there, the default minNodes decides whether an autoscaled pool keeps working.

The PR description still says the PR pins v0.1.0-bf0e690d until provider-coreweave has a release, and could say it pins v0.1.0.

Reviewed by modelplane-review ad611b9. Reply, or react 👍 or 👎, on each comment to help tune it.

Comment thread docs/content/platform/telemetry.md Resolved
@negz

negz commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

A second run in US-EAST-04A, with a CPU system pool (turin-gp, since the org has no cd-gp-i64-erapids quota) and the GPU pool autoscaled from the start:

  • Every controller ran on the CPU system node, and the DCGM exporter and DRA kubelet plugin DaemonSets scheduled nothing there.
  • The cluster was Ready 27 minutes after I applied the InferenceCluster. The GPU node took another 26 minutes, waiting on CoreWeave capacity.
  • The autoscaler took the idle GPU pool to zero 23 minutes after I deleted the model. Redeploying the model didn't bring it back: the autoscaler reported pod didn't trigger scale-up: 1 cannot allocate all claims on every pass for as long as I watched. That's GPU pools can't scale to zero on clouds using vanilla cluster-autoscaler #345: CKS's autoscaler can't model a DRA claim against a pool with no nodes. The XRD's minNodeCount now says to keep it at 1 or higher.

The provider pin is now the released v0.1.0, which upstream says has no code changes from the build both runs used.

@negz

negz commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator Author

/review

@modelplane-review modelplane-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes since my last review look close to ready, with a few docs points from the move to RTX PRO 6000 and one test gap.

The new minNodes default settles the earlier thread on autoscaled pools reaching zero. What's left is mostly the getting started pages, which no longer fit the CoreWeave path in a few places.

The docs commit's message says the pages note that node pools appear only once the cluster is running, but this round removed that note.

Reviewed by modelplane-review 337445e. Reply, or react 👍 or 👎, on each comment to help tune it.

@negz
negz marked this pull request as ready for review November 6, 2026 23:57

@modelplane-review modelplane-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes since my last review raise nothing new, and the PR looks ready to merge.

Reviewed by modelplane-review 337445e. Reply, or react 👍 or 👎, on each comment to help tune it.

@dennis-upbound dennis-upbound left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Nic, this reads really well and follows the shape of the other cluster XRs closely. I like the Usage chain for teardown and keeping the kubeconfig's observed data when the token can't be read. The two-Object trick for targetNodes took me a minute, but the comment explains it well, and moving the Service settings into gateway_proxy is a nice general fix. A few small things inline, mostly API consistency and a docs note about the token. None of it blocks for me. LGTM.

Comment thread apis/inferenceclusters/definition.yaml Resolved
negz added 4 commits November 6, 2026 17:18
Modelplane can't provision clusters on CoreWeave. Teams there have to
create a CKS cluster themselves and register it with source Existing.

CoreWeaveCluster provisions a CKS cluster in a dedicated VPC using
provider-coreweave. CKS differs from the other clouds in two ways that
shape the function.

CKS has no cloud API for node pools. A pool is a NodePool resource
inside the cluster, so every pool, the system pool included, is a
provider-kubernetes Object composed once the cluster is Ready. The
Objects keep their Delete policy: removing a pool from the spec must
remove its nodes. compose-usages runs after the function to hold the
provider-kubernetes ProviderConfig until every Object is gone, and
Usages hold the kubeconfig Secret and the cluster until the
ProviderConfig is gone, so teardown can't strand the Objects without a
way to reach the cluster. Another Usage holds the VPC until the cluster
is gone: CoreWeave's API answers a request to delete a VPC a cluster
still uses with NotFound, which provider-coreweave takes to mean the
VPC is already gone, leaking it.

A pool's Object is Ready once CKS validates the pool rather than once
its nodes arrive, so autoscaling and node replacement don't flap the
cluster's readiness. The exception is the system pool until the
CoreWeaveCluster is first Ready: its Object waits for a node, so the
serving stack doesn't install onto a cluster with nowhere to run it.

CKS's managed autoscaler owns an autoscaled pool's targetNodes, and CKS
rejects a NodePool without it. One Object can't manage such a pool: it
would force its targetNodes back over the autoscaler's, and leaving the
field out of its manifest would remove it while the Object was its only
field manager. provider-kubernetes applies each Object under a field
manager named for the Object, so an autoscaled pool gets two. One
creates and deletes the NodePool with a seed targetNodes, and the
other, without targetNodes, keeps the rest of the pool current. CKS's
autoscaler can't scale a pool up from zero nodes when its pods' devices
bind through DRA, as Modelplane's do, so the seed is at least one node
and minNodes defaults to the seed. A pool reaches zero, and stays there,
only if minNodeCount asks for it.

CKS publishes no kubeconfig. Its managed API endpoint accepts the same
CoreWeave API access token the provider uses, behind a publicly trusted
certificate, so the function reads the token from the Secret the
CoreWeave ProviderConfig references and renders a kubeconfig from it
and the cluster's observed endpoint. This copies the token into a
Secret in the XR's namespace.

CoreWeave manages the GPU stack, so GPU pools carry the label that
keeps the managed GPU Operator's device plugin off them, leaving the
serving stack's DRA driver as the only allocator.

CKS provides a ReadWriteMany StorageClass, shared-vast, on every
cluster, but it retains a volume after its claim is deleted, which
would leave a deleted ModelCache's weights behind on billed storage.
The function composes modelplane-rwx-fs for ModelCache instead, a copy
of the observed shared-vast that deletes the volume. It's copied rather
than written out because its parameters name the org's storage tenant
and the zone.

Towards modelplaneai#94.

Signed-off-by: Nic Cope <nicc@rk0n.org>
An InferenceCluster can't provision a cluster on CoreWeave, even though
CoreWeaveCluster can now build one.

This adds CoreWeave as a cluster source and a provisioning provider.
An InferenceCluster with source CoreWeave composes a CoreWeaveCluster
from its zone, Kubernetes version, credentials and the pools' classes,
then wires the cluster's kubeconfig into a ServingStack like Vultr and
Civo do, with no identity layered on top. ModelCache gets the
StorageClass the CoreWeaveCluster reports.

The serving stack's CoreWeave half installs cert-manager, the NVIDIA
DRA driver and a DCGM exporter. CoreWeave runs a managed NVIDIA GPU
Operator and NFD on every CKS cluster, and documents that users mustn't
install a second GPU Operator. It also installs NVIDIA's DRA driver for
ComputeDomains only, so Modelplane installs it again with GPU
allocation on.

CoreWeave runs a DCGM exporter too, in cw-exporters, but it attributes
a GPU's series only to pods the device plugin allocated it to. The
serving stack installs NVIDIA's standalone exporter with DRA
attribution, as on Nebius, and the collector's GPU job skips
cw-exporters so it doesn't collect every GPU series twice. The two
exporters run side by side on a CKS node without disturbing each
other's profiling metrics.

Towards modelplaneai#94.

Signed-off-by: Nic Cope <nicc@rk0n.org>
CoreWeave Kubernetes Service gives a LoadBalancer Service a private
address unless the Service carries the
service.beta.kubernetes.io/coreweave-load-balancer-type: public
annotation. Both of Modelplane's gateways need public addresses: an
InferenceGateway reaches cluster gateways across clouds, and callers
reach the InferenceGateway from outside the cluster. Neither function
that composes a gateway knows which cloud it runs on.

A cloud half of the serving stack can now declare the Service settings
its load balancers need, as a subset of Envoy Gateway's
KubernetesServiceSpec. The serving stack builds the GatewayClass's
EnvoyProxy from them, beside the externalTrafficPolicy pin every cloud
gets. The cluster gateway uses that EnvoyProxy as is. The
InferenceGateway's own EnvoyProxy now sets mergeType StrategicMerge, so
Envoy Gateway merges it onto the GatewayClass's rather than replacing
it, and its Service gets the same settings and pin without the
function knowing the cloud. The CoreWeave half declares the public
annotation.

CKS assigns a LoadBalancer Service its address once, when Envoy Gateway
creates it. A Service created before the EnvoyProxy exists has no
public annotation, and keeps the private address it gets even after
the annotation arrives. The cluster Gateway now waits for the EnvoyProxy
to be Ready before it's first composed.

Towards modelplaneai#94.

Signed-off-by: Nic Cope <nicc@rk0n.org>
The docs list CoreWeave CKS as a planned provider with no Crossplane
provider, and say nothing about how to provision a cluster there.

This marks CKS as provisioning-supported, links provider-coreweave, and
adds CoreWeave to the getting started guide and the InferenceCluster
and InferenceClass concept pages, with manifests for each. They note
what differs from the other clouds: the CoreWeave API token also
authenticates Modelplane to the cluster.

Towards modelplaneai#94.

Signed-off-by: Nic Cope <nicc@rk0n.org>
@negz

negz commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator Author

A final run in US-EAST-04A followed the CoreWeave getting started guide end to end with its own manifests: two rtxp6000-8x clusters on turin-gp system nodes, the InferenceGateway, qwen-demo, then the scale steps with qwen-west pinned to the second cluster by clusterSelector. Completions went through the shared endpoint, and once both clusters were reachable the responses' model names showed traffic split across both deployments.

It found a race, now fixed: Envoy Gateway created the second cluster's gateway Service before the EnvoyProxy carrying CKS's public annotation existed, and CKS gave the Service a private address it kept after the annotation arrived. The cluster Gateway now waits for the EnvoyProxy to be Ready before it's first composed. A recreated cluster created the EnvoyProxy 16 seconds before the Gateway and got a public address on the first try.

The run also reproduced a VPC update loop in provider-coreweave v0.1.0 (upbound/provider-upjet-coreweave#6). The fix is in progress upstream in coreweave/terraform-provider-coreweave#554.

@negz
negz merged commit 9eb9487 into modelplaneai:main Nov 7, 2026
15 checks passed
haarchri added a commit to haarchri/modelplane that referenced this pull request Nov 17, 2026
UpCloud's load balancer needs - TCP frontend mode and names that survive
the cloud controller's 64-character truncation - were plumbed as
serviceAnnotations through the InferenceCluster status and ServingStack
spec, with a {gateway} template and a per-Gateway EnvoyProxy that
restated the externalTrafficPolicy pin. The annotation also broke an
InferenceGateway with spec.tls: its Service has two ports, and the
controller matches config frontends to ports by name, so the single
unnamed frontend left the load balancer one nameless frontend on port 0
with no backend.

Instead, a cloud half now declares what its load balancers need on every
gateway Service as a GATEWAY_SERVICE subset of Envoy Gateway's
KubernetesServiceSpec, which join() folds into the GatewayClass-level
EnvoyProxy. The inference gateway's EnvoyProxy merges onto it with
mergeType: StrategicMerge, names its Service inference-gateway on every
cloud - one code path, with only the annotation depending on the cloud -
and on UpCloud alone overrides the config annotation with a named TCP
frontend per listener. On UpCloud the name is also load-bearing: the
controller already prefixes every load balancer with the cluster UUID,
so a Service name that differs from the cluster gateway's within the
truncated remainder is enough; the UUID-built name, the {gateway}
template and UpCloudCluster's clusterId status all go. On every other
cloud the rename replaces Envoy Gateway's generated
envoy-modelplane-system-inference-gateway-<hash> Service, so an
existing gateway gets a new LoadBalancer Service, with a new load
balancer and address. This is the mechanism modelplaneai#655 introduced for
CoreWeave, so both clouds share one shape.

The autoscaler fork lists every node group in the cluster each refresh
and gives any group missing from its --nodes list a minimum of one node
and a maximum of the cluster plan's limit, so it could scale the system
group down to one node or a fixed GPU pool up on pending pods, fighting
Crossplane as it reset each back to its nodeCount. Whenever any pool
autoscales, the group list now also pins the system group and every
fixed pool at their node count. compose_autoscaler_secret takes the
Autoscaler compose() already narrowed, a test now covers the node-group
sentinel that keeps groups composed through a Ready regression on
clusters without an autoscaler, and the node pool docs stop suggesting a
System pool beside the injected one, note that Premium and Starter plans
ignore diskSizeGb, and drop the stale GPU-pools-only wording.

Works around UpCloudLtd/upcloud-cloud-controller-manager#14.

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
haarchri added a commit to haarchri/modelplane that referenced this pull request Nov 19, 2026
UpCloud's load balancer needs - TCP frontend mode and names that survive
the cloud controller's 64-character truncation - were plumbed as
serviceAnnotations through the InferenceCluster status and ServingStack
spec, with a {gateway} template and a per-Gateway EnvoyProxy that
restated the externalTrafficPolicy pin. The annotation also broke an
InferenceGateway with spec.tls: its Service has two ports, and the
controller matches config frontends to ports by name, so the single
unnamed frontend left the load balancer one nameless frontend on port 0
with no backend.

Instead, a cloud half now declares what its load balancers need on every
gateway Service as a GATEWAY_SERVICE subset of Envoy Gateway's
KubernetesServiceSpec, which join() folds into the GatewayClass-level
EnvoyProxy. The inference gateway's EnvoyProxy merges onto it with
mergeType: StrategicMerge, names its Service inference-gateway on every
cloud - one code path, with only the annotation depending on the cloud -
and on UpCloud alone overrides the config annotation with a named TCP
frontend per listener. On UpCloud the name is also load-bearing: the
controller already prefixes every load balancer with the cluster UUID,
so a Service name that differs from the cluster gateway's within the
truncated remainder is enough; the UUID-built name, the {gateway}
template and UpCloudCluster's clusterId status all go. On every other
cloud the rename replaces Envoy Gateway's generated
envoy-modelplane-system-inference-gateway-<hash> Service, so an
existing gateway gets a new LoadBalancer Service, with a new load
balancer and address. This is the mechanism modelplaneai#655 introduced for
CoreWeave, so both clouds share one shape.

The autoscaler fork lists every node group in the cluster each refresh
and gives any group missing from its --nodes list a minimum of one node
and a maximum of the cluster plan's limit, so it could scale the system
group down to one node or a fixed GPU pool up on pending pods, fighting
Crossplane as it reset each back to its nodeCount. Whenever any pool
autoscales, the group list now also pins the system group and every
fixed pool at their node count. compose_autoscaler_secret takes the
Autoscaler compose() already narrowed, a test now covers the node-group
sentinel that keeps groups composed through a Ready regression on
clusters without an autoscaler, and the node pool docs stop suggesting a
System pool beside the injected one, note that Premium and Starter plans
ignore diskSizeGb, and drop the stale GPU-pools-only wording.

Works around UpCloudLtd/upcloud-cloud-controller-manager#14.

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
haarchri added a commit that referenced this pull request Nov 19, 2026
* Add provider-upcloud to the project dependencies

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

* Add UpCloudCluster to provision UpCloud Managed Kubernetes clusters

Modelplane can't provision clusters on UpCloud. This adds a namespaced
UpCloudCluster XR and a compose-upcloud-cluster function that provision
a UKS cluster on a dedicated SDN network, with a fixed system node
group and a KubernetesNodeGroup per user pool. UKS has no inline pools,
so unlike Civo and Vultr the system pool is its own Ready-gated node
group. KubernetesNodeGroup has no name field, so the function pins each
group's external name to its pool name, keeping UKS node group names
deterministic.

The UKS kubeconfig embeds static client certificates, so the connection
secret alone reaches the cluster. fetchKubeconfig must be true or the
provider publishes no kubeconfig at all.

UKS has no server-side autoscaler and the cluster autoscaler's UpCloud
cloud provider is an unmerged upstream PR (kubernetes/autoscaler#10234),
so pools with maxNodeCount are scaled by UpCloud's fork image installed
through the upstream chart. UpCloud provisions no API credentials on
the cluster, so spec.autoscaler names a token Secret that an Object
copies to kube-system, resolved server-side through patchesFrom so the
token never passes through the function. The newest fork image is built
for Kubernetes 1.29, older than any UKS version this XR provisions; it
talks to the UpCloud API rather than version-skewed Kubernetes APIs,
but the skew is outside the autoscaler's support policy.

No ModelCache storage is composed: UpCloud File Storage is managed NFS
without a CSI driver, so it only supports statically provisioned
volumes.

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

* Provision InferenceClusters on UpCloud

An InferenceCluster could not be provisioned on UpCloud. This wires the
new UpCloudCluster XR into the platform: source: UpCloud on
InferenceCluster (zone, version, credentials and the autoscaler's token
Secret), an upcloud provisioning block on InferenceClass naming the GPU
server plan, and a hand-written UpCloud serving-stack half.

UKS GPU node images ship the NVIDIA driver and container toolkit at the
default root, so the half matches Nebius's shape: NFD and the DRA
driver, no GPU Operator. With no operator there is no DCGM exporter
either, so the modelplane_gpu_* and energy series are empty on UpCloud,
the same gap Nebius has.

The UKS kubeconfig embeds static client certificates, so the
ClusterProviderConfig carries no identity. No cache StorageClass is
relayed: UpCloud File Storage is NFS without a CSI driver. Fabric is
not mapped; UKS has no EFA or InfiniBand analog.

A CEL rule requires the upcloud.autoscaler Secret reference when a pool
sets maxNodeCount, failing at admission rather than on the composed
UpCloudCluster's own validation, where the user wouldn't see it.

The upcloud e2e leg fakes GPU nodes the way compose-upcloud-cluster
shapes them and audits the stack's placement on them.

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

* Document the UpCloud provider

UpCloud wasn't in the providers table or any of the provisioning docs.
This adds the UKS row and tabs across the platform and getting-started
pages, the UpCloud manifests they embed, and the API-reference example.
The getting-started path includes a model deployment, which the other
recent clouds still lack.

The docs state what UpCloud doesn't support: no ModelCache (File
Storage is NFS without a CSI driver, so only static volumes), no
fabric, no GPU metrics (no GPU Operator, so no DCGM exporter), and GPU
plans in fi-hel2 only with account-gated quota. Pools with maxNodeCount
need the autoscaler token Secret, which the concepts manifest shows how
to create.

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

* feat(e2e): disk size configurable, loadbalancer annotations

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

* feat(nodes): take care that system is reserved, take care of autoscaler

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

* feat(ss): add dcgm exporter and add tcp for lb

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

* feat(parity): update branch after #556 merged, role from accelerator, cpu pools, draDriver tags, xrds update for accelerator, e2e leg, tests

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

* Declare gateway Service settings in the serving stack's cloud half

UpCloud's load balancer needs - TCP frontend mode and names that survive
the cloud controller's 64-character truncation - were plumbed as
serviceAnnotations through the InferenceCluster status and ServingStack
spec, with a {gateway} template and a per-Gateway EnvoyProxy that
restated the externalTrafficPolicy pin. The annotation also broke an
InferenceGateway with spec.tls: its Service has two ports, and the
controller matches config frontends to ports by name, so the single
unnamed frontend left the load balancer one nameless frontend on port 0
with no backend.

Instead, a cloud half now declares what its load balancers need on every
gateway Service as a GATEWAY_SERVICE subset of Envoy Gateway's
KubernetesServiceSpec, which join() folds into the GatewayClass-level
EnvoyProxy. The inference gateway's EnvoyProxy merges onto it with
mergeType: StrategicMerge, names its Service inference-gateway on every
cloud - one code path, with only the annotation depending on the cloud -
and on UpCloud alone overrides the config annotation with a named TCP
frontend per listener. On UpCloud the name is also load-bearing: the
controller already prefixes every load balancer with the cluster UUID,
so a Service name that differs from the cluster gateway's within the
truncated remainder is enough; the UUID-built name, the {gateway}
template and UpCloudCluster's clusterId status all go. On every other
cloud the rename replaces Envoy Gateway's generated
envoy-modelplane-system-inference-gateway-<hash> Service, so an
existing gateway gets a new LoadBalancer Service, with a new load
balancer and address. This is the mechanism #655 introduced for
CoreWeave, so both clouds share one shape.

The autoscaler fork lists every node group in the cluster each refresh
and gives any group missing from its --nodes list a minimum of one node
and a maximum of the cluster plan's limit, so it could scale the system
group down to one node or a fixed GPU pool up on pending pods, fighting
Crossplane as it reset each back to its nodeCount. Whenever any pool
autoscales, the group list now also pins the system group and every
fixed pool at their node count. compose_autoscaler_secret takes the
Autoscaler compose() already narrowed, a test now covers the node-group
sentinel that keeps groups composed through a Ready regression on
clusters without an autoscaler, and the node pool docs stop suggesting a
System pool beside the injected one, note that Premium and Starter plans
ignore diskSizeGb, and drop the stale GPU-pools-only wording.

Works around UpCloudLtd/upcloud-cloud-controller-manager#14.

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

* Stop installing the Prometheus stack on UpCloud

the UpCloud half predates it and still carried the chart. The UpCloud
e2e leg caught the consequence: its node-exporter DaemonSet tolerated
both inference pool taints and landed on the fake GPU and CPU nodes.
It goes the way #624 took it - the chart, the e2e leg's node-exporter
fan-out expectation, and its key in the serving stack's UpCloud test
cases. GPU telemetry is unaffected: the collector scrapes
dcgm-exporter directly and never read this Prometheus.

The gateway's LoadBalancer Service name is now pinned on every cloud
rather than only on UpCloud, and UpCloud's frontend list derives from
the same listener list the Gateway composes, so a listener added later
reaches both. The gateway-level load balancer config replaces the
class-level JSON value whole, so each side now points at the other for
a key added to one. New tests pin what nothing covered: the UpCloud
entry in the stacks' GATEWAY_SERVICE table, and the autoscaler pinning
a fixed pool beside an autoscaled one.

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

* Update the Vale styles hash for new upstream releases

The style packages resolve to GitHub releases/latest zips, so the
fixed-output hash breaks whenever one publishes a release. Rerun with
the hash Nix reports, as the derivation's comment prescribes.

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>

---------

Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support CoreWeave Kubernetes Service (CKS) for cluster provisioning

2 participants