Cloud client retries nothing: a 502/503/504 on an idempotent request fails the whole operation #137

Part of #90 (PR #61 catch-up), recovered by the triage correction.

State on main

src/ndi/cloud/client.py contains no retry or backoff of any kind, and is byte-identical to its version at the fork point ed78f8b:

$ a=$(git show ed78f8b:src/ndi/cloud/client.py | git hash-object --stdin)
$ b=$(git show origin/main:src/ndi/cloud/client.py | git hash-object --stdin)
$ [ "$a" = "$b" ] && echo IDENTICAL
IDENTICAL

$ git show origin/main:src/ndi/cloud/client.py | grep -ic 'retry\|backoff'
0

Every request is a single attempt. A transient gateway error anywhere in a long multi-document sync fails that call, and the operation above it with it.

Why it bites here specifically

The API sits behind an API Gateway whose Lambda cap is 29 s; exceeding it surfaces as a 504, not as a slow success. That is a routine outcome on large documents, not an exceptional one — and a plain retry of an idempotent request usually succeeds once the backend catches up.

The shape of the fix

catchup/ndi-python-2026-08 has it on the client, which is the right layer — one place, every call site covered:

RETRY_STATUSES = frozenset({502, 503, 504})
RETRY_METHODS  = frozenset({"GET", "HEAD", "OPTIONS", "PUT", "DELETE"})
MAX_RETRIES    = 3      # total attempts = 1 + retries on a retryable response
RETRY_BACKOFF  = 0.5    # seconds; grows linearly per attempt

with the retry inside _request, around the self._session.request(...) call.

The method allow-list is the part worth preserving deliberately: POST is excluded because most POST routes create resources, so a retry could duplicate them. That is a correctness constraint, not a tuning knob — if a POST route is genuinely idempotent it should be opted in explicitly and individually.

Two things worth improving over the branch version while porting:

• Backoff is linear (0.5 * attempt) with no jitter. Exponential with jitter is the safer default when many clients hit the same recovering backend. Small change, worth making now.
• A retried-and-still-failed request should say so. The branch's version raises the same CloudAPIError whether it tried once or three times; including the attempt count makes a genuine outage distinguishable from a slow route in the logs.

Also consider whether the connection-level exception path (requests.RequestException, currently an immediate CloudAPIError) should retry too — a dropped connection on a GET is as transient as a 503. The branch does not retry it.

Verification

A test with a stubbed transport asserting: a 504 on a GET is retried and succeeds; a 504 on a POST is not retried; a persistent 504 raises after exactly MAX_RETRIES attempts; and a 4xx is never retried.