diff --git a/content/en/documentation/for-adopters/control-plane/_index.md b/content/en/documentation/for-adopters/control-plane/_index.md index fe994d8..9122d5d 100644 --- a/content/en/documentation/for-adopters/control-plane/_index.md +++ b/content/en/documentation/for-adopters/control-plane/_index.md @@ -388,6 +388,19 @@ EDC has an eventing system that code can plug into and receive events when somet EDC implements reliable messaging for all interactions, so it is important to understand how this quality of service works. First, all messages have a unique ID and are idempotent. If a particular message is not acknowledged, it will be resent. Therefore, it is expected the receiving endpoint will perform de-duplication (which all EDC components do). Second, reliable messaging works across restarts. For example, if a runtime crashes before it can send a response, the response will be sent either by another instance (if running in a cluster) or by the runtime when it comes back online. Reliability is achieved by recording the state of all interactions using state machines to a transactional store such as Postgres. State transitions are initiated in the context of a transaction by sending a message to the counterparty, which is only committed after an acknowledgment is received. +How the counter-party answers a message decides what happens next: + +- **2xx:** the message is acknowledged, and the state machine moves on. +- **4xx:** the message is rejected for good, and the negotiation or transfer process is terminated. As required by the + Dataspace Protocol, a connector answers `400` to a message that is not valid in the current state of the process, and + `404` when the process does not exist or the client is not authorized (catalog requests get `401`). +- **5xx, or no answer** (timeout, unreachable counter-party): the message is sent again with a growing delay + (`edc.negotiation.send.retry.*`, `edc.transfer.send.retry.*`, see [Tuning](../tuning/_index.md#state-machines)), until + the retry limit is reached. A connector answers `503` when it can't process a message now but may later: the + negotiation or transfer process is being modified, e.g. by an earlier copy of the same message, or the token of the + counter-party couldn't be verified because its credential service or DID document was not reachable. The copy that + arrives after the first one was processed is recognized by its message id and acknowledged. + ## Transfer Processes After a Contract Negotiation has been finalized, a consumer can request data associated with an asset by initiating a *Transfer Process* via the Management API. diff --git a/content/en/documentation/for-adopters/control-plane/state-machines/_index.md b/content/en/documentation/for-adopters/control-plane/state-machines/_index.md index d7d98ab..150327d 100644 --- a/content/en/documentation/for-adopters/control-plane/state-machines/_index.md +++ b/content/en/documentation/for-adopters/control-plane/state-machines/_index.md @@ -46,6 +46,12 @@ lists the accepted ones: what happens, when, the impact, and how to mitigate it - **State changed before a step that can fail.** `processProviderInitial` transitions the transfer process before storing the data address of the data plane; a retryable failure of that store would make the retry transition fail. It is not reachable with `VaultDataAddressStore`, whose failures are fatal. +- **Lost response of a data plane signal.** When the response to a `prepare` or `start` signal is lost (e.g. it times + out) after the data plane created the data flow, the signal is sent again, by the HTTP client or by the state + machine. As defined by Data Plane Signaling, the data plane rejects it for the existing data flow, and the transfer + process fails while its data flow exists. Each attempt also selects the data plane again: with several data planes + matching the transfer, a retry may go to another one. Mitigation: have slow data planes answer asynchronously, and + keep `edc.http.client.timeout.read` above the time a data plane takes to answer. ## Poll-based state machine diff --git a/content/en/documentation/for-adopters/data-plane/_index.md b/content/en/documentation/for-adopters/data-plane/_index.md index 3c31512..daae1fc 100644 --- a/content/en/documentation/for-adopters/data-plane/_index.md +++ b/content/en/documentation/for-adopters/data-plane/_index.md @@ -73,6 +73,13 @@ A data plane can also answer asynchronously, and report the outcome later throug response to events. For example, the control plane policy monitor could send a TERMINATE message if a policy violation is encountered. +A signal that can't be delivered (timeout, connection error), or that the data plane answers with a 5xx, is sent again +by the transfer process state machine, with the delays and the limit of `edc.transfer.send.retry.*`. A signal the data +plane rejects with a 4xx fails the transfer process. A data plane that needs long to prepare or start a data flow should +answer asynchronously: when the response to `prepare` or `start` is lost, e.g. because it takes longer than +`edc.http.client.timeout.read` (30 s), the signal is sent again, and the data plane rejects it for the data flow it +already created (see [State Machines Known Limitations](../control-plane/state-machines/_index.md#common-to-both-state-machines)). + ### Securing the Signaling API The data plane notifies the control plane of the progress of a transfer (prepared, started, completed, errored) by diff --git a/content/en/documentation/for-contributors/data-plane/data-plane-signaling/_index.md b/content/en/documentation/for-contributors/data-plane/data-plane-signaling/_index.md index ca08c81..8f8283c 100644 --- a/content/en/documentation/for-contributors/data-plane/data-plane-signaling/_index.md +++ b/content/en/documentation/for-contributors/data-plane/data-plane-signaling/_index.md @@ -111,7 +111,7 @@ transfer process is sent as the `profile`, together with the `labels` and the `m | Transfer process step | Side | DPS call | |---|---|---| | INITIAL | consumer | `POST {endpoint}/prepare` | -| STARTING | provider | `POST {endpoint}/start` | +| INITIAL | provider | `POST {endpoint}/start` | | STARTED received from the provider | consumer | `POST {endpoint}/{flowId}/started` | | SUSPENDING | both | `POST {endpoint}/{flowId}/suspend` | | RESUMING | both | `POST {endpoint}/{flowId}/resume` | @@ -125,6 +125,12 @@ A data plane answers `prepare`, `start` and `resume` either synchronously, with case, the transfer process waits in `PREPARATION_REQUESTED` or `STARTUP_REQUESTED` until the data plane notifies the control plane. +A signal fails with `ERROR_RETRY` when it couldn't be delivered (timeout, I/O error), when the data plane answers with a +5xx, or when the authorization profile couldn't obtain a token, and the transfer process state machine sends it again +with its backoff and retry limit (`edc.transfer.send.retry.*`). A 4xx answer fails with `FATAL_ERROR`, which ends the +step: e.g. a failed `prepare` terminates the consumer transfer process, a failed `start` makes the provider terminate +it. + ### 4.2 Notifications from the data plane The data plane notifies the control plane through the Signaling API, a dedicated API context