Skip to content

fix(healthchecks): detect failing HPAs regardless of condition order - #277

Open
lbarrosby wants to merge 1 commit into
crossplane-contrib:mainfrom
lbarrosby:fix/hpa-health-failed-conditions
Open

lbarrosby wants to merge 1 commit into
crossplane-contrib:mainfrom
lbarrosby:fix/hpa-health-failed-conditions

Conversation

@lbarrosby

Copy link
Copy Markdown

Description of your changes

The HPA health check could never see a failing HPA.

It compared condition types against FailedGetScale, FailedUpdateScale, FailedGetResourceMetric and InvalidSelector. Those are reasons. The HPA controller reports them on an AbleToScale or ScalingActive condition with status False, which is also how gitops-engine (the source this check cites) matches them. So the degraded branch was dead code, and the result came down to whether ScalingActive or ScalingLimited was True, and in which order the conditions were listed, which is what #169 points out.

This shows up in practice. When the metrics an HPA reads go away (metrics-server down, a KEDA or external metric failing), the controller sets ScalingActive=False with a FailedGet*Metric reason and doesn't touch ScalingLimited, which is often still True/TooFewReplicas from an earlier sync. The function reported that HPA as ready.

The check now looks for a failure on every condition first, matching type, status and reason, and only then for a healthy condition, so the order no longer matters:

  • AbleToScale=False with FailedGetScale or FailedUpdateScale
  • ScalingActive=False with any reason except ScalingDisabled (target scaled to zero on purpose)

Backoff (AbleToScale=False, BackoffBoth/BackoffUpscale/BackoffDownscale) is normal stabilization and is not treated as a failure.

The old failure tests used the reasons as condition types, so they passed without exercising anything. I rewrote them with conditions as the HPA controller writes them, and added a failing metric next to a stale ScalingLimited=True in both orders, plus backoff and scaling disabled. Four of the new cases fail on main.

The other checks mentioned in the issue don't have this problem: Job's Complete and Failed can't both be True, and the ReplicaSet and Deployment checks already look at each condition on its own.

Fixes #169

I have:

go test ./... and golangci-lint run ./healthchecks/ are clean.

The HPA check compared condition types against FailedGetScale,
FailedUpdateScale, FailedGetResourceMetric and InvalidSelector. Those are
reasons, not types: the HPA controller reports them on an AbleToScale or
ScalingActive condition whose status is False. The degraded branch could
never match, so the result depended only on whether ScalingActive or
ScalingLimited was True.

That reports a broken HPA as ready. When the metrics an HPA reads stop
being available, the controller sets ScalingActive to False with a
FailedGet*Metric reason and leaves ScalingLimited as it was, often True
with TooFewReplicas from an earlier sync.

Check every condition for a failure first, matching on type, status and
reason, and only then look for a healthy one, so the order of the
conditions no longer matters. Backing off (AbleToScale False with a
Backoff* reason) and ScalingDisabled are not failures.

Fixes crossplane-contrib#169

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@lbarrosby

Copy link
Copy Markdown
Author

Friendly ping. If this isn't something you want in the project, feel free to say so and I'll close it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concerned about healthcheck condition checks

1 participant