Skip to content

Fix InferenceCluster InvalidNodePool and ClusterProviderConfig bugs - #604

Closed
RedbackThomson wants to merge 2 commits into
modelplaneai:mainfrom
RedbackThomson:inference-cluster-condition-fixes
Closed

RedbackThomson wants to merge 2 commits into
modelplaneai:mainfrom
RedbackThomson:inference-cluster-condition-fixes

Conversation

@RedbackThomson

Copy link
Copy Markdown
Contributor

Description of your changes

Towards #603.

Two bugs in compose-inference-cluster that the stage conditions in #603 build on.

When a node pool's class had no provisioning block for the cluster's cloud, the builder set ClusterReady=False with reason InvalidNodePool and returned, but a later ClusterReady=Provisioning on the same reconcile replaced it, so only a Warning event said what was wrong. The early return also left the cloud cluster XR out of desired state, so Crossplane would try to delete a running cluster whose shared class lost its block. The check is now a fatal gate beside check_devices_supported, before any cloud is composed, naming every offending class.

The kubeconfig ClusterProviderConfig was composed only while the cloud cluster XR was Ready. One node group or Helm release going unready on a running cluster dropped it from desired state while every Object on the cluster still used it. It's now kept for as long as it's observed, the way the ServingStack already is.

I have:

  • Read and followed Modelplane's contribution process.
  • Run nix flake check (or ./nix.sh flake check) and made sure it passes.
  • Added or updated tests covering any composition function changes.
  • Signed off every commit with git commit -s.

Each compose_<cloud>_cluster builder checked that every node pool's
class had a provisioning block for the cloud. On a miss it set
ClusterReady=False with reason InvalidNodePool and returned, but
compose_<cloud> carried on to derive_conditions, whose later
ClusterReady=Provisioning replaced it. Only a Warning event said what
was wrong. The early return also left the cloud cluster XR out of
desired state, so Crossplane would try to delete a running cluster
whose shared class lost its block.

The check is now a gate beside check_devices_supported, before any
cloud is composed. Like that gate it names every offending class and
returns a fatal result, so the condition survives and Crossplane holds
the cluster's resources rather than collecting them.

Towards modelplaneai#603.

Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
compose-inference-cluster composed the ClusterProviderConfig that
reaches a provisioned cluster only while the cloud cluster XR was
Ready. A cloud cluster XR is unready whenever any resource it composes
is, so one node group or Helm release going unready on a running
cluster dropped the ClusterProviderConfig from desired state, and
Crossplane would delete it while every Object on the cluster still
used it: the namespaces, PoolObservations, ModelReplicas and
ModelRoutes.

Once composed, the ClusterProviderConfig is now kept for as long as it
is observed, the way the ServingStack already is.

Towards modelplaneai#603.

Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
@RedbackThomson

Copy link
Copy Markdown
Contributor Author

Folded into #607.

@RedbackThomson
RedbackThomson deleted the inference-cluster-condition-fixes branch October 30, 2026 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant