Repository navigation
Conversation
The HPA check compared condition types against FailedGetScale, FailedUpdateScale, FailedGetResourceMetric and InvalidSelector. Those are reasons, not types: the HPA controller reports them on an AbleToScale or ScalingActive condition whose status is False. The degraded branch could never match, so the result depended only on whether ScalingActive or ScalingLimited was True. That reports a broken HPA as ready. When the metrics an HPA reads stop being available, the controller sets ScalingActive to False with a FailedGet*Metric reason and leaves ScalingLimited as it was, often True with TooFewReplicas from an earlier sync. Check every condition for a failure first, matching on type, status and reason, and only then look for a healthy one, so the order of the conditions no longer matters. Backing off (AbleToScale False with a Backoff* reason) and ScalingDisabled are not failures. Fixes crossplane-contrib#169 Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
lbarrosby
requested review from
negz,
sergenyalcin,
turkenf and
ulucinar
as code owners
November 7, 2026 16:17
Author
|
Friendly ping. If this isn't something you want in the project, feel free to say so and I'll close it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
The HPA health check could never see a failing HPA.
It compared condition types against
FailedGetScale,FailedUpdateScale,FailedGetResourceMetricandInvalidSelector. Those are reasons. The HPA controller reports them on anAbleToScaleorScalingActivecondition with statusFalse, which is also how gitops-engine (the source this check cites) matches them. So the degraded branch was dead code, and the result came down to whetherScalingActiveorScalingLimitedwasTrue, and in which order the conditions were listed, which is what #169 points out.This shows up in practice. When the metrics an HPA reads go away (metrics-server down, a KEDA or external metric failing), the controller sets
ScalingActive=Falsewith aFailedGet*Metricreason and doesn't touchScalingLimited, which is often stillTrue/TooFewReplicasfrom an earlier sync. The function reported that HPA as ready.The check now looks for a failure on every condition first, matching type, status and reason, and only then for a healthy condition, so the order no longer matters:
AbleToScale=FalsewithFailedGetScaleorFailedUpdateScaleScalingActive=Falsewith any reason exceptScalingDisabled(target scaled to zero on purpose)Backoff (
AbleToScale=False,BackoffBoth/BackoffUpscale/BackoffDownscale) is normal stabilization and is not treated as a failure.The old failure tests used the reasons as condition types, so they passed without exercising anything. I rewrote them with conditions as the HPA controller writes them, and added a failing metric next to a stale
ScalingLimited=Truein both orders, plus backoff and scaling disabled. Four of the new cases fail onmain.The other checks mentioned in the issue don't have this problem: Job's
CompleteandFailedcan't both beTrue, and the ReplicaSet and Deployment checks already look at each condition on its own.Fixes #169
I have:
go test ./...andgolangci-lint run ./healthchecks/are clean.