waitForDS() can report success prematurely during statefulset rolling updates

[m4um4u1]

We had an issue with the wait command when waiting for DS replicas:

waitForDS() {
    local ds=$1
  
    waitResourceExists "sts/${ds}"
    local rep=$($K_CMD $NAMESPACE_OPT get sts $ds -o=jsonpath='{.spec.replicas}')
    ((rep--))
    message "rep=${rep}" "debug"
    for i in $(seq 0 $rep) ; do
      waitForResource ready "pod/${ds}-${i}"
    done
  }

This function checks individual pods for readiness, but does not check whether they have actually been updated. Additionally, it iterates in the wrong order (low to high), while statefulsets by default restart the pods from the highest number to the lowest.

This means the command can report success prematurely: the lower numbered pod hasn't been restarted yet (so it's still "ready" from the previous deployment), while the higher numbered pod has already been restarted and the wait command correctly waited for it.

Example:
In a cluster with two ds replicas

1. The rollout terminates ds-*-1 first (highest number).
2. The wait command checks ds-*-0 which is still ready from the previous deployment, so it passes immediately.
3. The wait command checks ds-*-1 and waits for the new pod to become ready.
4. The wait command reports success.
5. The rollout now terminates ds-*-0 to update it, causing a brief period where other components need to realize the downtime.

In our case, this caused flaky amster imports in our deploy pipeline because amster was started right after the wait command succeeded, but the old and ready ds pod was just terminated.

We where able to get around this issue with the following:

waitForDS() {
  local ds=$1
  waitForRollout "statefulset/${ds}"
}
  
waitForRollout() {
  local resource=$1
  echo "Waiting for ${resource} rollout to complete."
  if waitResourceExists "$resource" ; then
    kube rollout status ${resource} --timeout="${TIMEOUT}s"
  else
    echo "ERROR: $resource not found. Skipping."
  fi
}

This waits for the full rollout to complete (all pods updated to the new version and ready), regardless of order.

The question is of course, should it wait for the full rollout to complete (safer), or just for at least one pod to be ready (faster, sufficient for basic platform availability)?
For am and idm it waits for the deployments, why not for the statefulset for ds?
What was the intention here?

[lee-baines]

Hi @m4um4u1, thanks for highlighting.

This function was really designed for the initial deployment hence the reason it checks low to high on the pod index number.
But I agree, this would be a nice improvement. I'll create a backlog item to take a look.

[m4um4u1]

Hello @lee-baines,
Thank you for your answer and looking into this.

This function is used by the wait command. That would mean the wait command is as well intended for a fresh cluster, do I understand this right?

[lee-baines]

Yes correct @m4um4u1, it was designed to help stage a new deployment and to allow testing of each component during deployment as part of a CICD pipeline. But your suggestion looks like it would be a nice additional to the functionality

[m4um4u1]

@lee-baines thank you for the explanation. We are also using this in CICD pipelines. We often deploy feature branches on our test clusters where we have a pipeline for building then deploying the new images.
In this pipeline we first build the images then deploy them, and then wait for the components to be ready again with the wait command. When everything is ready (according to the command) we do import dynamic config with amster, and then run E2E tests.