Skip to content

Report provisioning stages from the cloud cluster XRs - #606

Closed
RedbackThomson wants to merge 5 commits into
modelplaneai:mainfrom
RedbackThomson:cluster-stage-conditions
Closed

RedbackThomson wants to merge 5 commits into
modelplaneai:mainfrom
RedbackThomson:cluster-stage-conditions

Conversation

@RedbackThomson

Copy link
Copy Markdown
Contributor

Description of your changes

Towards #603.

While it provisioned, a cloud cluster XR reported nothing beyond its own Ready, which stays False until every resource it composes is Ready. A cluster still creating its control plane looked the same as one finishing its last GPU node pool, or one whose node group had failed.

Each compose-<cloud>-cluster function now splits what it composes into the user's node pools and the cluster (network, IAM, control plane, system nodes, add-ons), and reports ClusterReady, NodePoolsReady and status.nodePools from them. The logic is the same on all six clouds; only the resources grouped under each pool differ.

A stage or pool is Failed while the provider reports a resource's LastAsyncOperation False, with the provider's error, prefixed with the resource name, as the message. Synced=False isn't treated as a failure, because a managed resource is also Synced=False while it waits on a reference, which is routine while a cluster provisions. Only upjet providers set LastAsyncOperation, so a failing Helm release or provider-kubernetes Object stays Provisioning. upjet also clears that condition on the next poll after a failed create that left an ID, so Failed lasts only as long as the provider reports it.

ClusterReady's Provisioning message is a count (12 of 52 resources ready). It changes as resources finish, and every ModelDeployment, ModelCache, InferenceGateway, ModelRoute and MetricMapping requires every InferenceCluster, so each change reconciles them. compose-gke-cluster also reports ClusterReady=Provisioning while it waits for the GCP ProviderConfig it reads the project from.

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes.
  • Added or updated tests covering any composition function changes.
  • Signed off every commit with git commit -s.

Each compose_<cloud>_cluster builder checked that every node pool's
class had a provisioning block for the cloud. On a miss it set
ClusterReady=False with reason InvalidNodePool and returned, but
compose_<cloud> carried on to derive_conditions, whose later
ClusterReady=Provisioning replaced it. Only a Warning event said what
was wrong. The early return also left the cloud cluster XR out of
desired state, so Crossplane would try to delete a running cluster
whose shared class lost its block.

The check is now a gate beside check_devices_supported, before any
cloud is composed. Like that gate it names every offending class and
returns a fatal result, so the condition survives and Crossplane holds
the cluster's resources rather than collecting them.

Towards modelplaneai#603.

Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
compose-inference-cluster composed the ClusterProviderConfig that
reaches a provisioned cluster only while the cloud cluster XR was
Ready. A cloud cluster XR is unready whenever any resource it composes
is, so one node group or Helm release going unready on a running
cluster dropped the ClusterProviderConfig from desired state, and
Crossplane would delete it while every Object on the cluster still
used it: the namespaces, PoolObservations, ModelReplicas and
ModelRoutes.

Once composed, the ClusterProviderConfig is now kept for as long as it
is observed, the way the ServingStack already is.

Towards modelplaneai#603.

Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
Provisioning a cluster spends most of its time building node pools,
but neither the cloud cluster XRs nor InferenceCluster report anything
per pool: the only signal is the XR's own Ready, False until every
pool and every other composed resource is done.

Each cloud cluster XR, and InferenceCluster, gets a status.nodePools
list keyed by pool name. A pool's state is Pending until its cloud
resources are composed, Provisioning until their provider reports them
Ready, then Ready, or Failed while the provider reports that its last
operation on one of them failed, with the provider's error as the
message. The system pool is left out because it is part of the base
cluster rather than a pool the user asked for.

Towards modelplaneai#603.

Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
While it provisioned, a cloud cluster XR reported nothing beyond its
own Ready, which stays False until every resource it composes is
Ready. A cluster still creating its control plane looked the same as
one finishing its last GPU node pool, or one whose node group had
failed to create.

Each compose-<cloud>-cluster function now splits what it composes into
the user's node pools and the cluster: everything else, such as the
network, control plane, system nodes and add-ons. ClusterReady is True
once every cluster resource is marked ready, and NodePoolsReady once
ClusterReady is and every pool's resources are. status.nodePools
reports each pool as Pending until its resources are composed, then
Provisioning, then Ready.

A stage or pool is Failed while the provider reports a resource's
LastAsyncOperation False, with the provider's error, prefixed with the
resource's name, as the message. Synced=False doesn't count as a
failure, because a managed resource is also Synced=False while it
waits on a reference to resolve, which is routine while a cluster
provisions. Only upjet providers set LastAsyncOperation, so a Helm
release or provider-kubernetes Object that fails stays Provisioning.

ClusterReady's Provisioning message counts the cluster's resources that
are ready, such as "12 of 52 resources ready". compose-gke-cluster also
reports ClusterReady while it waits for the GCP ProviderConfig it reads
the project from, rather than composing nothing and reporting nothing.

Towards modelplaneai#603.

Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown

Docs preview: https://modelplane-docs-pr-606.vercel.app (ready once the site's Content workflow finishes)

@RedbackThomson

Copy link
Copy Markdown
Contributor Author

Folded into #607.

@RedbackThomson
RedbackThomson deleted the cluster-stage-conditions branch October 30, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant