Repository navigation
Conversation
|
Docs preview: https://modelplane-docs-pr-607.vercel.app (ready once the site's Content workflow finishes) |
|
Since we're touching this, it occurs to me that "backend" is old terminology for the serving stack. Could we rename the condition to These days "backend" mostly means a |
503468e to
53461db
Compare
|
/review |
1 similar comment
|
/review |
There was a problem hiding this comment.
I think this is close. The stage model follows #603, and what's left are questions about how the stages behave outside first provisioning.
The main ones are whether the fallback for a cloud cluster XR that doesn't report the new conditions should exist at all, and what a stage that already finished should report when an earlier one goes False on a running cluster. The others cover which failures report Failed, how the stage boundaries meet #550, and some test structure.
functions/compose-inference-cluster/tests/test_fn.py line 3403: Nit: the observed EKSCluster is now written out inline in four cases, three of them new. CONTRIBUTING asks for a helper once a resource appears in three or more cases, and an observed resource gets one separate from the desired one, such as _observed_eks_cluster(*, conditions=..., node_pools=...).
Reviewed by modelplane-review d7f2116. Reply, or react 👍 or 👎, on each comment to help tune it.
53461db to
dd68374
Compare
69062a6 to
608645e
Compare
compose-inference-cluster composed the ClusterProviderConfig that reaches a provisioned cluster only while the cloud cluster XR was Ready. A cloud cluster XR is unready whenever any resource it composes is, so one node group or Helm release going unready on a running cluster dropped the ClusterProviderConfig from desired state, and Crossplane would delete it while every Object on the cluster still used it: the namespaces, PoolObservations, ModelReplicas and ModelRoutes. Once composed, the ClusterProviderConfig is now kept for as long as it is observed, the way the ServingStack already is. Towards modelplaneai#603. Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
Provisioning a cluster spends most of its time building node pools, but neither the cloud cluster XRs nor InferenceCluster report anything per pool: the only signal is the XR's own Ready, False until every pool and every other composed resource is done. Each cloud cluster XR, and InferenceCluster, gets a status.nodePools list keyed by pool name. A pool's state is Pending until its cloud resources are composed, Provisioning until their provider reports them Ready, then Ready, or Failed while the provider reports that its last operation on one of them failed, with the provider's error as the message. The system pool is left out because it is part of the base cluster rather than a pool the user asked for. Towards modelplaneai#603. Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
While it provisioned, a cloud cluster XR reported nothing beyond its own Ready, which stays False until every resource it composes is Ready. A cluster still creating its control plane looked the same as one finishing its last GPU node pool, or one whose node group had failed to create. Each compose-<cloud>-cluster function now splits what it composes into the node pools and the cluster: everything else, such as the network, control plane, add-ons and system nodes, including any role: System pool. ClusterReady is True once every cluster resource is marked ready, and NodePoolsReady once every pool's resources are. NodePoolsReady is Pending on ClusterReady only while no pool is Ready, so on a running cluster the pools keep reporting their own state when the cluster or a new pool provisions. status.nodePools reports each pool as Pending until its resources are composed, then Provisioning, then Ready. A stage or pool is Failed while the provider reports a resource's LastAsyncOperation False, with the provider's error, prefixed with the resource's name, as the message. Synced=False doesn't count as a failure, because a managed resource is also Synced=False while it waits on a reference to resolve, which is routine while a cluster provisions. upjet sets LastAsyncOperation only on resources it creates asynchronously, so a failed IAM role, Helm release or provider-kubernetes Object stays Provisioning. ClusterReady's Provisioning message counts the cluster's resources that are ready, such as "12 of 52 resources ready". compose-gke-cluster also reports ClusterReady while it waits for the GCP ProviderConfig it reads the project from, rather than composing nothing and reporting nothing. Towards modelplaneai#603. Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
InferenceCluster reported ClusterReady and BackendReady, with no messages. ClusterReady covered the whole cloud cluster XR, node pools included, so it read Provisioning for most of a cluster's creation and said nothing about which part was slow or had failed. InferenceCluster now reports Accepted, ClusterReady, NodePoolsReady and ServingStackReady. While a new cluster provisions, each is Pending until the one before it is True, so the first condition that isn't True names the stage it's in. Once a stage has started it reports its own state, so a running cluster's ServingStackReady stays True while a new node pool provisions. Accepted carries the checks that fail the reconcile, such as WaitingForClasses, InvalidNodePool and UnsupportedDevices, which reported on ClusterReady. While it's False, a stage that hasn't been composed is Pending, and a running cluster's stages keep reporting their own state. ClusterReady, NodePoolsReady and status.nodePools are relayed from the cloud cluster XR as it reports them, including the Unknown status Crossplane gives them when the cloud cluster XR's function fails, while the XR's Ready stays True. An Existing cluster reports both stages with the reason Existing and has no status.nodePools. ClusterRunning becomes Available. BackendReady becomes ServingStackReady, because "backend" now means a ModelEndpoint a gateway routes to, and its BackendHealthy, Installing and WaitingForCluster reasons become Available, Provisioning and Pending. Nothing in Modelplane reads either condition. Ready, and the status fields the scheduler and other functions read, are unchanged. kubectl get ic shows each stage's reason in CLUSTER, POOLS and STACK columns. Towards modelplaneai#603. Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
The getting started guide told readers to wait on Ready while a cluster provisioned, with no way to see how far it had got. The InferenceCluster page now describes the stage conditions, their reasons and status.nodePools, and the guide points to it from the provisioning step. Fixes modelplaneai#603. Signed-off-by: Nicholas Thomson <RedbackThomson@users.noreply.github.com>
608645e to
09df6a1
Compare
Description of your changes
Fixes #603.
This implements the stage conditions proposed in #603. Each
compose-<cloud>-clusterfunction reportsClusterReady,NodePoolsReadyandstatus.nodePoolson its own XR, with the same logic on all eight clouds; only the resources grouped under each pool differ.compose-inference-clusterrelays them as the cloud cluster XR reports them and derivesAcceptedandServingStackReady. A stage isPendingon an earlier one only until it's composed. After that it reports its own state, so on a running cluster the serving stack staysAvailablewhile a new node pool provisions orAcceptedfails.Breaking for anything reading conditions:
ClusterRunningbecomesAvailable, andBackendReadybecomesServingStackReady, because "backend" now means aModelEndpointa gateway routes to. ItsBackendHealthy,InstallingandWaitingForClusterreasons becomeAvailable,ProvisioningandPending.ClusterReadyno longer covers node pools. Nothing in Modelplane reads either condition, andReadyis unchanged.The first commit fixes a bug this builds on. The kubeconfig
ClusterProviderConfigwas composed only while the cloud cluster XR was Ready, so one unready node group dropped it while every Object on the cluster still used it; it's now kept while it's observed. The checks that fail the reconcile (missing classes or reservations, invalid node pools, unsupported devices) now report onAcceptedrather thanClusterReady.I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.