Make retries and failover safe for requests that change state

CHANGELOG.md

@@ -7,15 +7,37 @@ and the project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.
 
 ## [Unreleased]
 
+### Added
+
+- `ConflictError` for HTTP 409.
+- The `max_retry_after` option (`OCTA_MAX_RETRY_AFTER`): the longest
+  `Retry-After` the client waits for, 60 seconds by default.
+- `Retry-After` given as an HTTP date.
+
 ### Changed
 
+- HTTP 400 raises `ValidationError` instead of `APIError`.
+- GET, PUT and DELETE requests are retried after connection errors and
+  timeouts, not only after error responses.
+- `ConfigurationError` is also a `ValueError`; invalid `OCTA_*` numbers raise
+  it.
+- Any transport failure raises `APIConnectionError` or `APITimeoutError`
+  instead of an `httpx` exception.
+
 - **Breaking:** Python 3.11 or later is required; Python 3.10 is no longer
   supported.
 
 ### Removed
 
 - **Breaking:** the `octaspace-smoke` command and `python -m octaspace`.
 
+### Fixed
+
+- With several API hosts, a request that changes something (starting or
+  stopping a session, setting prices) is no longer sent to another host after
+  a timeout, so that it cannot run twice.
+- The API token no longer appears in `repr()` of `ClientConfig`.
+
 ## [0.1.0] - 2026-04-14
 
 ### Added

docs/errors-and-reliability.md

@@ -27,11 +27,12 @@ from the API raises a subclass of `APIError`:
 | `AuthenticationError` | 401 | The API token is missing, wrong or regenerated |
 | `APIPermissionError` | 403 | Access denied |
 | `NotFoundError` | 404 | Unknown session, or a node that is not yours |
-| `ValidationError` | 422 | A parameter is invalid |
+| `ValidationError` | 400, 422 | The request was rejected: a parameter is invalid, or a session cannot be started (for example an insufficient balance) |
+| `ConflictError` | 409 | The request conflicts with the current state |
 | `RateLimitError` | 429 | Too many requests |
 | `ServerError` | 5xx | A temporary problem on the API side |
 | `BadGatewayError`, `ServiceUnavailableError`, `GatewayTimeoutError` | 502, 503, 504 | Subclasses of `ServerError` |
-| `APIError` | any other | For example 400 when a session cannot be started (insufficient balance) or prices are invalid |
+| `APIError` | any other | Base class of all of the above |
 
 When the API cannot be reached at all, the error is a `NetworkError`:
 `APIConnectionError` (DNS, connection or TLS failure) or `APITimeoutError`.
@@ -48,6 +49,7 @@ OctaSpaceError
     ├── AuthenticationError
     ├── APIPermissionError
     ├── NotFoundError
+    ├── ConflictError
     ├── ValidationError
     ├── RateLimitError
     └── ServerError
@@ -62,11 +64,11 @@ An `APIError` has the HTTP `status_code`, the `request_id` (or `None`) and the
 full `response`. Its message is the API's reason when the response has one:
 
 ```python
-from octaspace import APIError
+from octaspace import ValidationError
 
 try:
     client.services.mr.create(node_id=312, image="ubuntu:24.04", disk_size=20)
-except APIError as exc:
+except ValidationError as exc:
     print(f"status {exc.status_code}: {exc}")  # status 400: Insufficient balance
     print(exc.response.data)
 ```
@@ -92,22 +94,22 @@ except RateLimitError as exc:
 
 ## Retries
 
-GET requests are retried automatically on the statuses 408, 429, 500, 502,
-503 and 504, up to `max_retries` times (default 2), with exponential backoff
-and a little jitter. When the response has a `Retry-After` header, the SDK
-waits that many seconds instead. After the last attempt the error is raised.
+GET requests are retried automatically on connection errors, timeouts and the
+statuses 408, 429, 500, 502, 503 and 504, up to `max_retries` times (default
+2), with exponential backoff and a little jitter. When the response has a
+`Retry-After` header, the SDK waits that long instead; when the API asks to
+wait longer than `max_retry_after` (default 60 seconds), the `RateLimitError`
+is raised at once. After the last attempt the error is raised.
 
 > [!IMPORTANT]
 > Starting a session (`create()`), stopping it (`stop()`), setting prices and
-> generating a wallet are never retried, so that they cannot run twice.
-> Decide yourself whether to repeat them after an error. `nodes.reboot()` is a
-> GET request and is retried like the others.
+> generating a wallet are retried only when the connection could not be
+> established, so that they cannot run twice. After a timeout or an error
+> response, decide yourself whether to repeat them. `nodes.reboot()` is a GET
+> request and is retried like the others.
 
-A connection error or a timeout is not retried on the same host. With
-[several API hosts](#several-api-hosts) the request moves to the next host.
-
-Set the number of retries and the backoff with `max_retries`,
-`retry_interval` and `backoff_factor`; see
+Set the number of retries and the waits with `max_retries`, `retry_interval`,
+`backoff_factor` and `max_retry_after`; see
 [Client configuration](getting-started.md#options).
 
 ## Several API hosts
@@ -127,12 +129,9 @@ with OctaSpaceClient(
     client.nodes.list()
 ```
 
-Requests are spread over the hosts in turn. When a host cannot be reached or
-times out, the request is sent to the next host, and the failed host is left
-out for 30 seconds. An HTTP error response, including 5xx, does not move the
-request to another host.
-
-> [!WARNING]
-> This also applies to `create()` and `stop()`. A request that timed out may
-> still have reached the API: before you start a session again, check
-> `client.sessions.list()`.
+Requests are spread over the hosts in turn. When a host cannot be reached, the
+request is sent to the next host, and the failed host is left out for 30
+seconds. A GET request also moves to the next host after a timeout. A request
+that changes something (`create()`, `stop()`, setting prices) moves only when
+the connection could not be established, so that it cannot run twice. An HTTP
+error response, including 5xx, does not move the request to another host.

docs/getting-started.md

@@ -40,7 +40,8 @@ finally:
     client.close()
 ```
 
-An unknown or invalid option raises `octaspace.ConfigurationError`.
+An unknown or invalid option raises `octaspace.ConfigurationError`, a
+subclass of `ValueError`.
 
 ## Options
 
@@ -54,6 +55,7 @@ An unknown or invalid option raises `octaspace.ConfigurationError`.
 | `write_timeout` | `30.0` | `OCTA_WRITE_TIMEOUT` | Seconds to wait while sending a request |
 | `max_retries` | `2` | `OCTA_MAX_RETRIES` | Retries of failed requests that are safe to repeat; `0` turns them off. See [Retries](errors-and-reliability.md#retries) |
 | `retry_interval` / `backoff_factor` | `0.5` / `2.0` | `OCTA_RETRY_INTERVAL` / `OCTA_BACKOFF_FACTOR` | Backoff between retries: `retry_interval × backoff_factor^attempt` seconds |
+| `max_retry_after` | `60.0` | `OCTA_MAX_RETRY_AFTER` | Longest `Retry-After` the SDK waits for; a longer wait raises the error at once |
 | `keep_alive` | `False` | `OCTA_KEEP_ALIVE` | Reuse connections, see [below](#keep-alive-connections) |
 | `pool_size` | `5` | `OCTA_POOL_SIZE` | Connections kept open with `keep_alive` |
 | `max_connections` | `20` | `OCTA_MAX_CONNECTIONS` | Most connections open at the same time |
@@ -82,7 +84,7 @@ with OctaSpaceClient(config=config) as client:
 
 Keyword arguments to `from_env()` override the environment:
 `ClientConfig.from_env(max_retries=0)`. Boolean variables accept `1`, `true`,
-`yes` and `on`.
+`yes` and `on`. An invalid number raises `ConfigurationError`.
 
 ## Shared client
 

docs/resources.md

@@ -65,8 +65,8 @@ uuid = response.data["uuid"]
 
 The session is created at once; its container is still being set up. Check
 `is_ready` with [`info()`](#session-details). A rejected request, for example
-for an insufficient balance, raises `octaspace.APIError` with status 400 and
-the reason as its message.
+for an insufficient balance, raises `octaspace.ValidationError` with the
+reason as its message.
 
 ## VPN
 
@@ -208,7 +208,7 @@ client.nodes.update_prices(312, base=25, storage=2, traffic=1)
 Pass one or more of `base`, `storage` and `traffic`, each an integer from 0
 to 10000; see `node_prices` in the
 [API reference](https://api.octa.space/api-docs) for the units. Any other
-field or value raises `APIError` with status 400.
+field or value raises `ValidationError`.
 
 ### Reboot
 

src/octaspace/__init__.py

@@ -13,6 +13,7 @@
     AuthenticationError,
     BadGatewayError,
     ConfigurationError,
+    ConflictError,
     GatewayTimeoutError,
     NetworkError,
     NotFoundError,
@@ -92,6 +93,7 @@ def shutdown_shared_client() -> None:
     "BadGatewayError",
     "ClientConfig",
     "ConfigurationError",
+    "ConflictError",
     "GatewayTimeoutError",
     "NetworkError",
     "NotFoundError",

src/octaspace/_transport/httpx_transport.py

@@ -6,10 +6,10 @@
 import httpx
 
 from ..config import ClientConfig
-from ..errors import APIConnectionError, APITimeoutError, raise_for_response
+from ..errors import APIConnectionError, APITimeoutError, NetworkError, raise_for_response
 from ..hooks import HookDispatcher
 from ..response import APIResponse
-from .retry import compute_backoff, is_retryable_response
+from .retry import can_resend, compute_backoff, is_retryable_response
 from .rotator import UrlRotator
 
 
@@ -34,36 +34,26 @@ def request(
         body: Any = None,
         headers: dict[str, str] | None = None,
     ) -> APIResponse:
-        normalized_method = method.upper()
-        normalized_path = path if path.startswith("/") else f"/{path}"
-        response: APIResponse | None = None
-        attempts = self._config.max_retries + 1
-        for attempt in range(attempts):
-            response = self._send_with_failover(
-                normalized_method,
-                normalized_path,
-                params=params,
-                body=body,
-                headers=headers,
-            )
-            if (
-                is_retryable_response(normalized_method, response.status_code)
-                and attempt < attempts - 1
-            ):
-                delay = compute_backoff(
-                    attempt=attempt,
-                    retry_interval=self._config.retry_interval,
-                    backoff_factor=self._config.backoff_factor,
-                    retry_after=response.retry_after,
+        method = method.upper()
+        path = path if path.startswith("/") else f"/{path}"
+        attempt = 0
+        while True:
+            try:
+                response = self._send_with_failover(
+                    method, path, params=params, body=body, headers=headers
                 )
+            except httpx.TransportError as exc:
+                if attempt >= self._config.max_retries or not can_resend(method, exc):
+                    raise _network_error(exc) from exc
+                time.sleep(self._backoff(attempt))
+            else:
+                delay = self._retry_delay(method, response, attempt)
+                if delay is None:
+                    if response.is_error:
+                        raise_for_response(response)
+                    return response
                 time.sleep(delay)
-                continue
-            break
-
-        assert response is not None
-        if response.is_error:
-            raise_for_response(response)
-        return response
+            attempt += 1
 
     def get(
         self,
@@ -132,6 +122,25 @@ def transport_stats(self) -> dict[str, Any]:
             stats["url"] = self._config.urls[0]
         return stats
 
+    def _retry_delay(self, method: str, response: APIResponse, attempt: int) -> float | None:
+        """Seconds to wait before repeating the request, or ``None`` not to repeat it."""
+        if attempt >= self._config.max_retries:
+            return None
+        if not is_retryable_response(method, response.status_code):
+            return None
+        retry_after = response.retry_after
+        if retry_after is not None and retry_after > self._config.max_retry_after:
+            return None
+        return self._backoff(attempt, retry_after=retry_after)
+
+    def _backoff(self, attempt: int, *, retry_after: int | None = None) -> float:
+        return compute_backoff(
+            attempt=attempt,
+            retry_interval=self._config.retry_interval,
+            backoff_factor=self._config.backoff_factor,
+            retry_after=retry_after,
+        )
+
     def _send_with_failover(
         self,
         method: str,
@@ -141,31 +150,17 @@ def _send_with_failover(
         body: Any = None,
         headers: dict[str, str] | None = None,
     ) -> APIResponse:
-        ordered_urls = self._ordered_urls()
-        last_error: Exception | None = None
-        for base_url in ordered_urls:
+        """Send to the next healthy host; move on to another one only when that is safe."""
+        *fallbacks, last = self._ordered_urls()
+        for base_url in fallbacks:
             try:
-                response = self._send_to_url(
-                    base_url,
-                    method,
-                    path,
-                    params=params,
-                    body=body,
-                    headers=headers,
+                return self._send_to_url(
+                    base_url, method, path, params=params, body=body, headers=headers
                 )
-            except (APIConnectionError, APITimeoutError) as exc:
-                last_error = exc
-                if self._rotator is not None:
-                    self._rotator.mark_failed(base_url)
-                continue
-
-            if self._rotator is not None:
-                self._rotator.mark_success(base_url)
-            return response
-
-        if last_error is not None:
-            raise last_error
-        raise APIConnectionError("All API endpoints unavailable")
+            except httpx.TransportError as exc:
+                if not can_resend(method, exc):
+                    raise
+        return self._send_to_url(last, method, path, params=params, body=body, headers=headers)
 
     def _send_to_url(
         self,
@@ -177,20 +172,20 @@ def _send_to_url(
         body: Any = None,
         headers: dict[str, str] | None = None,
     ) -> APIResponse:
-        request_headers = self._default_headers(headers)
-        url = self._build_url(base_url, path)
         try:
             response = self._client.request(
                 method,
-                url,
+                self._build_url(base_url, path),
                 params=params,
                 json=body,
-                headers=request_headers,
+                headers=self._default_headers(headers),
             )
-        except httpx.TimeoutException as exc:
-            raise APITimeoutError(str(exc)) from exc
-        except httpx.NetworkError as exc:
-            raise APIConnectionError(str(exc)) from exc
+        except httpx.TransportError:
+            if self._rotator is not None:
+                self._rotator.mark_failed(base_url)
+            raise
+        if self._rotator is not None:
+            self._rotator.mark_success(base_url)
         return self._hooks.response_wrapper(response)
 
     def _ordered_urls(self) -> list[str]:
@@ -217,3 +212,9 @@ def _default_headers(self, headers: dict[str, str] | None = None) -> dict[str, s
     @staticmethod
     def _build_url(base_url: str, path: str) -> str:
         return f"{base_url.rstrip('/')}/{path.lstrip('/')}"
+
+
+def _network_error(error: httpx.TransportError) -> NetworkError:
+    if isinstance(error, httpx.TimeoutException):
+        return APITimeoutError(str(error))
+    return APIConnectionError(str(error))