Repository navigation
Provision InferenceClusters on UpCloud - #594
Conversation
|
Docs preview: https://modelplane-docs-pr-594.vercel.app (ready once the site's Content workflow finishes) |
dennis-upbound
left a comment
There was a problem hiding this comment.
compose-upcloud-cluster: removing spec.autoscaler orphans a running autoscaler (should-fix)
The ProviderConfigs, the token Object and the Release are composed only while spec.autoscaler is set. When a user turns their last autoscaled pool into a fixed pool and drops autoscaler (which the CEL rule then allows), all three leave desired state. The Release and Object exclude Delete. The autoscaler keeps running with its old group list, and its node counts fight the nodeCount Crossplane now enforces on those pools. compose_cluster_autoscaler's docstring already handles this for the last pool going fixed. Composing the sub-block when spec.autoscaler or "release-cluster-autoscaler" in observed would cover this case the same way, with an empty group list.
Pool named system (should-fix)
On this one I'd reserve the name with CEL. The PR body says the other clouds can't hit it, but EKS and Nebius key their groups nodegroup-{name} next to nodegroup-system, and GKE uses nodepool-system, so they have the same collision. One rule on InferenceCluster spec.nodePools[].name would cover every cloud. Here the external name is also system, so a declared system pool would take over the system group in UKS as well.
Merge order with #556 (FYI)
_node_pool and compose_upcloud_cluster hard-code role="GPU". Once #556 merges, UpCloud needs the same changes the other six clouds got there: the role derived from accelerator, draDrivers tags on its stack and CPU pools.
Before ready: the checklist is unchecked, and Tested has no live UKS run yet (the e2e-clouds (upcloud) leg is KWOK).
|
found one issue in the e2e run: UpCloudLtd/upcloud-cloud-controller-manager#14 |
|
because of the LB issue - i updated the code to set annotations to override the default behaviour until UpCloudLtd/upcloud-cloud-controller-manager#3 gets shipped, disk size configurable like the other clouds because the default is to low for models and added dcgm exporter |
4e580c9 to
9ebd199
Compare
dennis-upbound
left a comment
There was a problem hiding this comment.
The code looks good to me, and you fixed both points from my earlier comments. Admission now rejects pools named system, and the function keeps composing the autoscaler release while it's observed, so removing spec.autoscaler doesn't orphan it. Thanks for the live UKS run in the description too.
I'd like these cleaned up before I approve:
- The PR description is out of date. It still says UpCloud has no DCGM exporter, but
stacks/clouds/upcloud.pyinstalls one now. The "Known rough edge" paragraph leaves reservingsystemopen for review, and both XRDs now have CEL rules for it. - Some commits use
feat(...)prefixes, which CONTRIBUTING.md asks us not to use. Two of the older commit bodies also still say there's no DCGM exporter. A reword or squash would fix both.
|
@dennis-upbound updated description we can Squash during merge ?! |
|
/review |
There was a problem hiding this comment.
This looks like it needs another look at two things before it's close: which node groups the UpCloud autoscaler touches, and the load balancer frontends on a gateway with more than one port.
The cluster XR, the UpCloud serving-stack half and leaving out ModelCache follow the pattern Civo and Vultr set, and I didn't find anything that conflicts with a committed design. The design questions are about the gateway serviceAnnotations plumbing and the autoscaler fork. The gateway change alters how every cloud composes its gateways, and it has no issue behind it, which CONTRIBUTING asks for before an API change.
The PR description doesn't mention the serviceAnnotations fields, the per-Gateway EnvoyProxy or the TCP frontend override, and those are the parts that need review judgment.
Reviewed by modelplane-review 5455585. Reply, or react 👍 or 👎, on each comment to help tune it.
9ebd199 to
4c7ee2f
Compare
|
Heads up: #655 (CoreWeave) needed per-cloud LoadBalancer Service annotations too, and solves it a bit differently from In #655 a cloud half of the serving stack declares the Service settings its load balancers need, as a subset of Envoy Gateway's For UpCloud, the TCP frontend mode looks like it could be declared the same way. The per-gateway LB name is the part that doesn't fit, since it depends on the cluster UUID. Giving each gateway's Service a distinct name ( |
099c45e to
b2e6856
Compare
|
/review |
There was a problem hiding this comment.
This looks close. The gateway rework and autoscaler pinning answer the earlier threads. The open design question is whether compose-inference-gateway should know about UpCloud. Beyond that, this PR and #655 need to settle who merges the gateway code they share, and two new behaviours lack tests.
Moving onto #655's mechanism dropped serviceAnnotations, the {gateway} template and status.clusterId. I traced the merged EnvoyProxy, the Service port names and the cloud controller's load balancer naming through Envoy Gateway v1.8.4 and the UpCloud cloud controller, and they match what the code expects.
Reviewed by modelplane-review 3017043. Reply, or react 👍 or 👎, on each comment to help tune it.
|
e2e test done: and the response: |
|
/review |
There was a problem hiding this comment.
This looks close. The new commit addresses the earlier threads on the gateway's cloud branching and the missing tests. What's left is two wording points and how the commit and description present the change.
All three affected functions' tests pass at this commit. Nothing in the functions, e2e or docs looks up the inference gateway's generated Service name or uses the removed UpCloud Prometheus.
The commit bundles the Prometheus removal, the gateway rework and two new tests under one subject, where CONTRIBUTING asks authors to merge each into the commit it belongs to. The Tested pod listing in the description still shows the Prometheus stack this PR no longer installs.
Reviewed by modelplane-review 3017043. Reply, or react 👍 or 👎, on each comment to help tune it.
|
@haarchri could you bump the provider to v1.1.0 which was released today? We've bumped the underlying TF provider to the latest version and resolved all existing CVEs |
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Modelplane can't provision clusters on UpCloud. This adds a namespaced UpCloudCluster XR and a compose-upcloud-cluster function that provision a UKS cluster on a dedicated SDN network, with a fixed system node group and a KubernetesNodeGroup per user pool. UKS has no inline pools, so unlike Civo and Vultr the system pool is its own Ready-gated node group. KubernetesNodeGroup has no name field, so the function pins each group's external name to its pool name, keeping UKS node group names deterministic. The UKS kubeconfig embeds static client certificates, so the connection secret alone reaches the cluster. fetchKubeconfig must be true or the provider publishes no kubeconfig at all. UKS has no server-side autoscaler and the cluster autoscaler's UpCloud cloud provider is an unmerged upstream PR (kubernetes/autoscaler#10234), so pools with maxNodeCount are scaled by UpCloud's fork image installed through the upstream chart. UpCloud provisions no API credentials on the cluster, so spec.autoscaler names a token Secret that an Object copies to kube-system, resolved server-side through patchesFrom so the token never passes through the function. The newest fork image is built for Kubernetes 1.29, older than any UKS version this XR provisions; it talks to the UpCloud API rather than version-skewed Kubernetes APIs, but the skew is outside the autoscaler's support policy. No ModelCache storage is composed: UpCloud File Storage is managed NFS without a CSI driver, so it only supports statically provisioned volumes. Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
An InferenceCluster could not be provisioned on UpCloud. This wires the new UpCloudCluster XR into the platform: source: UpCloud on InferenceCluster (zone, version, credentials and the autoscaler's token Secret), an upcloud provisioning block on InferenceClass naming the GPU server plan, and a hand-written UpCloud serving-stack half. UKS GPU node images ship the NVIDIA driver and container toolkit at the default root, so the half matches Nebius's shape: NFD and the DRA driver, no GPU Operator. With no operator there is no DCGM exporter either, so the modelplane_gpu_* and energy series are empty on UpCloud, the same gap Nebius has. The UKS kubeconfig embeds static client certificates, so the ClusterProviderConfig carries no identity. No cache StorageClass is relayed: UpCloud File Storage is NFS without a CSI driver. Fabric is not mapped; UKS has no EFA or InfiniBand analog. A CEL rule requires the upcloud.autoscaler Secret reference when a pool sets maxNodeCount, failing at admission rather than on the composed UpCloudCluster's own validation, where the user wouldn't see it. The upcloud e2e leg fakes GPU nodes the way compose-upcloud-cluster shapes them and audits the stack's placement on them. Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
UpCloud wasn't in the providers table or any of the provisioning docs. This adds the UKS row and tabs across the platform and getting-started pages, the UpCloud manifests they embed, and the API-reference example. The getting-started path includes a model deployment, which the other recent clouds still lack. The docs state what UpCloud doesn't support: no ModelCache (File Storage is NFS without a CSI driver, so only static volumes), no fabric, no GPU metrics (no GPU Operator, so no DCGM exporter), and GPU plans in fi-hel2 only with account-gated quota. Pools with maxNodeCount need the autoscaler token Secret, which the concepts manifest shows how to create. Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
0f6594a to
65eed54
Compare
|
/review |
|
/review |
There was a problem hiding this comment.
This looks close to ready. The design hasn't changed since my last review, and what's left is two nits and tidying the history.
The rebase onto #655 and the provider bump look right. The tests of compose-inference-cluster, compose-inference-gateway, compose-serving-stack and compose-upcloud-cluster pass at 65eed54. Between provider-upcloud v1.0.2 and v1.1.0, the only change to the CRDs compose-upcloud-cluster uses is the controller-gen annotation.
The fold into 275a2c9 looks partial. At that commit _upcloud_envoy_service holds the listener-building body, so it seems the EnvoyProxy's envoyService gets the listener list, and nothing calls _envoy_service. Could the gateway rework move whole into 275a2c9, and f2cc33a get back the opening line its body lost? The Vale hash commit repeats #678, already on main, so it can go. The PR description still says provider-upcloud v1.0.2 and that #655 hasn't merged.
Reviewed by modelplane-review 22e194c. Reply, or react 👍 or 👎, on each comment to help tune it.
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
…accelerator, cpu pools, draDriver tags, xrds update for accelerator, e2e leg, tests Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
UpCloud's load balancer needs - TCP frontend mode and names that survive
the cloud controller's 64-character truncation - were plumbed as
serviceAnnotations through the InferenceCluster status and ServingStack
spec, with a {gateway} template and a per-Gateway EnvoyProxy that
restated the externalTrafficPolicy pin. The annotation also broke an
InferenceGateway with spec.tls: its Service has two ports, and the
controller matches config frontends to ports by name, so the single
unnamed frontend left the load balancer one nameless frontend on port 0
with no backend.
Instead, a cloud half now declares what its load balancers need on every
gateway Service as a GATEWAY_SERVICE subset of Envoy Gateway's
KubernetesServiceSpec, which join() folds into the GatewayClass-level
EnvoyProxy. The inference gateway's EnvoyProxy merges onto it with
mergeType: StrategicMerge, names its Service inference-gateway on every
cloud - one code path, with only the annotation depending on the cloud -
and on UpCloud alone overrides the config annotation with a named TCP
frontend per listener. On UpCloud the name is also load-bearing: the
controller already prefixes every load balancer with the cluster UUID,
so a Service name that differs from the cluster gateway's within the
truncated remainder is enough; the UUID-built name, the {gateway}
template and UpCloudCluster's clusterId status all go. On every other
cloud the rename replaces Envoy Gateway's generated
envoy-modelplane-system-inference-gateway-<hash> Service, so an
existing gateway gets a new LoadBalancer Service, with a new load
balancer and address. This is the mechanism modelplaneai#655 introduced for
CoreWeave, so both clouds share one shape.
The autoscaler fork lists every node group in the cluster each refresh
and gives any group missing from its --nodes list a minimum of one node
and a maximum of the cluster plan's limit, so it could scale the system
group down to one node or a fixed GPU pool up on pending pods, fighting
Crossplane as it reset each back to its nodeCount. Whenever any pool
autoscales, the group list now also pins the system group and every
fixed pool at their node count. compose_autoscaler_secret takes the
Autoscaler compose() already narrowed, a test now covers the node-group
sentinel that keeps groups composed through a Ready regression on
clusters without an autoscaler, and the node pool docs stop suggesting a
System pool beside the injected one, note that Premium and Starter plans
ignore diskSizeGb, and drop the stale GPU-pools-only wording.
Works around UpCloudLtd/upcloud-cloud-controller-manager#14.
Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
the UpCloud half predates it and still carried the chart. The UpCloud e2e leg caught the consequence: its node-exporter DaemonSet tolerated both inference pool taints and landed on the fake GPU and CPU nodes. It goes the way modelplaneai#624 took it - the chart, the e2e leg's node-exporter fan-out expectation, and its key in the serving stack's UpCloud test cases. GPU telemetry is unaffected: the collector scrapes dcgm-exporter directly and never read this Prometheus. The gateway's LoadBalancer Service name is now pinned on every cloud rather than only on UpCloud, and UpCloud's frontend list derives from the same listener list the Gateway composes, so a listener added later reaches both. The gateway-level load balancer config replaces the class-level JSON value whole, so each side now points at the other for a key added to one. New tests pin what nothing covered: the UpCloud entry in the stacks' GATEWAY_SERVICE table, and the autoscaler pinning a fixed pool beside an autoscaled one. Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
The style packages resolve to GitHub releases/latest zips, so the fixed-output hash breaks whenever one publishes a release. Rerun with the hash Nix reports, as the derivation's comment prescribes. Signed-off-by: Christopher Haar <christopher.haar@upbound.io>
65eed54 to
48ab82c
Compare
Description of your changes
Modelplane couldn't provision clusters on UpCloud. This adds source: UpCloud to InferenceCluster, backed by a new namespaced UpCloudCluster XR and a compose-upcloud-cluster function on crossplane-contrib/provider-upcloud v1.0.2. An UpCloudCluster provisions a UKS cluster on a dedicated SDN network (UKS requires one per cluster, DHCP on, default route off), a fixed system node group, and one KubernetesNodeGroup per pool. UKS has no inline pools, so unlike Civo and Vultr the system pool is itself a Ready-gated composed resource, and KubernetesNodeGroup has no name field, so each group's external name is pinned to its pool name to keep UKS node group names deterministic.
The cloud-specific decisions, and why:
Credentials. The UKS kubeconfig embeds static client certificates, so the ClusterProviderConfig carries no identity (same as Vultr and Civo). fetchKubeconfig: true is required or the provider publishes no kubeconfig to the connection secret. The provider's credentials Secret is JSON, {"token": ...} or {"username": ..., "password": ...}.
GPU stack. UKS GPU node images pre-install the NVIDIA driver and container toolkit at the default root, so the hand-written serving-stack half matches Nebius's shape: NFD plus the DRA driver, no GPU Operator.
Gateway load balancers. UpCloud's cloud controller hardcodes every load balancer frontend to HTTP mode, which can't carry the cluster gateway's TLS (the handshake is parsed as HTTP and the connection reset), and truncates load balancer names to -- at 64 characters with no unique suffix, so both gateways' generated Service names collide and every load balancer after the first fails with RESOURCE_ALREADY_EXISTS (UpCloudLtd/upcloud-cloud-controller-manager#14). The serving stack's cloud half declares what UpCloud needs on every gateway's LoadBalancer Service as a GATEWAY_SERVICE subset of Envoy Gateway's KubernetesServiceSpec, folded into the GatewayClass-level EnvoyProxy; the inference gateway's EnvoyProxy merges onto it with mergeType: StrategicMerge and, on UpCloud alone, overrides the config annotation with a named TCP frontend per listener (the controller matches frontends to Service ports by name, and its single-frontend fallback can't cover the two ports a TLS gateway has) and names its Service so the truncated names stay distinct — the controller's own UUID prefix keeps them account-unique. This is the mechanism #655 introduces for CoreWeave, applied ahead of it so both clouds share one shape; no InferenceCluster or ServingStack API surface is added for it.
Autoscaling. UKS has no server-side autoscaler and the cluster autoscaler's UpCloud cloud provider is an unmerged upstream PR (kubernetes/autoscaler#10234), so pools with maxNodeCount are scaled by UpCloud's fork image (ghcr.io/upcloudltd/autoscaler) installed through the upstream chart with image/env overrides. UpCloud provisions no API credentials on the cluster, so spec.autoscaler names a token Secret that an Object copies to kube-system; provider-kubernetes resolves the copy server-side through patchesFrom, so the token never passes through the function. Two tradeoffs stated plainly: the newest fork image is built for Kubernetes 1.29, older than any UKS version this XR provisions (it talks to the UpCloud API rather than version-skewed Kubernetes APIs, but the skew is outside the autoscaler's support policy), and the fork enforces a minimum of one node per group, so no scale-to-zero. The fork also lists every node group in the cluster each refresh and gives any group missing from its --nodes list a minimum of one and a maximum of the cluster plan's limit, so whenever any pool autoscales the group list also pins the system group and every fixed pool at their node count, keeping the autoscaler off the groups Crossplane owns. A CEL rule requires the autoscaler Secret when any pool sets maxNodeCount, failing at admission rather than on the composed XR where the user wouldn't see it.
ModelCache. None. UpCloud File Storage is managed NFS without a CSI driver, so volumes can only be provisioned statically; the docs and cache_storage_class docstring name the gap.
Fabric. Not mapped; UKS has no EFA or InfiniBand analog.
Zones. GPU plans are currently fi-hel2 only, with account-gated quota; the docs say so.
No breaking changes; UpCloud is additive to the source, provider and cloud enums.
Fixes #
I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.Tested