Feature request: retry all 5xx responses on metric sample upload

[cypher7682]

Summary

retryWithBackoff only retries the S3 upload on HTTP 500 and 403. Any other 5xx — notably 503 Service Unavailable, which is a server-side response — falls straight through to break, is returned as an error, and the agent then calls log.Fatalf, terminating the process. A single transient 503 from the upload endpoint therefore crashes and restarts the pod, even though 503 is by definition a retryable, temporary condition, and arguably more temporary than 403.

We would like the retry logic to cover the whole 5xx range.

Observations

Logs

The container exited and was restarted by kubelet. The only two log lines were:

time="2026-09-15T05:19:08Z" level=warning msg="cloudability write failed: Request received 503 response"
time="2026-09-15T05:19:08Z" level=fatal msg="error sending metrics: Request received 503 response"

Both lines are the same error surfacing twice as it propagates — SendData logs it at warning, then the caller logs it at fatal and exits. Kubernetes recorded the termination as reason="Error" (non-zero exit), which is what alerted us.

Notably there were no retry log lines at all, i.e. the upload was abandoned on the first attempt rather than being retried up to MaxRetries.

To be clear about what this was and wasn't: a 503 is a genuine HTTP response received from the endpoint, so this was not an egress/connectivity problem on our side. It was a transient server-side unavailability that we would have expected the agent to ride out.

Root cause

In client/client.go, inside retryWithBackoff:

Line 258 — the retry predicate only matches 500 and 403:

if resp.StatusCode == http.StatusInternalServerError || resp.StatusCode == http.StatusForbidden {
    time.Sleep(getSleepDuration(i))
    log.Errorf("Put S3 Retry %d: Failed to put data to S3, Status: %s X-Amzn-Requestid: %s", i,
        statusMessage, awsRequestID)
    ...
    continue
}
log.Infof("Put S3 Retry %d: Successfully put data to S3, X-Amzn-Requestid: %s", i, awsRequestID)
break

Anything not matched hits break, and the response is returned to SendMetricSample, which fails it at line 178:

if resp.StatusCode != http.StatusOK {
    return fmt.Errorf("Request received %v response", resp.StatusCode)
}

That error reaches kubernetes/kubernetes.go, where it is logged at warning (line 555) and then fatally (line 477):

log.Fatalf("error sending metrics: %v", err)

Result: one transient 503 → process exit → pod restart.

It is worth highlighting the asymmetry: had the endpoint returned 500, the agent would have backed off and retried. Because it returned 503 — the status that most explicitly means "temporarily unavailable" — it exited immediately. This seems contrary from the intent of the status code.

And, one more highlight, GetUploadURL already retries on any non-200 response.

Requested change

Widen the retry predicate at client/client.go:258 to cover all 5xx, while retaining the existing 403 case:

// Retry all 5xx server errors (e.g. 500, 502, 503, 504) as well as
// 403, which pre-signed S3 upload URLs can return transiently.
if resp.StatusCode >= http.StatusInternalServerError || resp.StatusCode == http.StatusForbidden {