diff --git a/src/content/docs/fundamentals/reference/connection-limits.mdx b/src/content/docs/fundamentals/reference/connection-limits.mdx index a20551bd85b..819452f62e7 100644 --- a/src/content/docs/fundamentals/reference/connection-limits.mdx +++ b/src/content/docs/fundamentals/reference/connection-limits.mdx @@ -19,6 +19,14 @@ When HTTP/HTTPS traffic is [proxied through Cloudflare](/fundamentals/concepts/h ## Between Cloudflare and origin server +### What connection means here + +A **TCP connection** is established when its TCP handshake completes. For a new **TCP-based HTTPS connection**, setup also requires SSL/TLS negotiation. TCP can therefore be established while HTTPS setup remains unfinished. + +Requests can reuse established connections, and a proxied request may traverse multiple connections. A connection timeout does not, by itself, identify which operation timed out or what caused it. Reused connections and QUIC do not have the same per-request TCP/TLS setup sequence as a new TCP-based HTTPS connection. + +The TCP establishment and request-acknowledgment rows describe specific TCP timeout conditions. They are not an exhaustive list of every operation that can produce [522](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-522/). Neither value is a universal TCP-plus-TLS setup budget. A timeout while TLS remains pending can be reported as `522` rather than [525](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-525/). + :::note If you are using [Cloudflare tunnels](/cloudflare-one/networks/connectors/cloudflare-tunnel/), refer to [Origin configuration](/cloudflare-one/networks/connectors/cloudflare-tunnel/configure-tunnels/origin-parameters/) to view or modify your connection settings. @@ -26,8 +34,8 @@ If you are using [Cloudflare tunnels](/cloudflare-one/networks/connectors/cloudf | Type | Limit (seconds) | HTTP status code at limit | [Configurable](/fundamentals/reference/connection-limits/#configurable-limits) | | ----------------------------------------------------------------------------------------- | --------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ | -| Complete TCP Connection | 19 | [522](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-522/) | No | -| TCP ACK Timeout | 90 | [522](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-522/) | No | +| TCP connection establishment | 19 | [522](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-522/) | No | +| TCP request acknowledgment timeout | 90 | [522](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-522/) | No | | TCP Keep-Alive Interval | 30 | [520](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-520/) | No | | Proxy Idle Timeout | 900 | [520](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-520/) | No | | Proxy Read Timeout | 125 | [524](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-524/) | [Yes, for Enterprise zones](/api/resources/zones/subresources/settings/methods/edit/) | diff --git a/src/content/docs/support/troubleshooting/http-status-codes/4xx-client-error/error-499.mdx b/src/content/docs/support/troubleshooting/http-status-codes/4xx-client-error/error-499.mdx index 8061e67ee53..421913c975e 100644 --- a/src/content/docs/support/troubleshooting/http-status-codes/4xx-client-error/error-499.mdx +++ b/src/content/docs/support/troubleshooting/http-status-codes/4xx-client-error/error-499.mdx @@ -7,6 +7,8 @@ products: - support --- +import { Steps } from "~/components"; + ## 499 Client Close Request Cloudflare records a 499 status code in logs and analytics when a client closes the connection before the request is complete. No 499 HTTP response is sent to the client — the code only appears in logs. @@ -21,54 +23,67 @@ Common patterns that produce elevated 499 rates include: | Pattern | Signals to look for | Starting point | |---|---|---| -| **Client deadline shorter than path latency** | Rate elevated on specific endpoints; matched completed requests on those endpoints show elevated `OriginResponseDurationMs` or longer edge times | Compare matched completed requests for the same endpoint and window; check origin-side timing | -| **Cache miss increasing latency** | Cache hit rate dropped for affected paths; 499 rate tracks the hit-rate change | Check cache configuration for affected paths | -| **Possible edge or network issue** | `OriginIP` absent; `CacheCacheStatus` is MISS or BYPASS; rate notably elevated in one data center relative to matched cohorts elsewhere; no origin-side or cache explanation | Collect evidence across matched cohorts, then contact Cloudflare Support if unresolved | +| **Client deadline shorter than path latency** | Rate elevated on specific endpoints. Matched completed requests on those endpoints show elevated `OriginResponseDurationMs` or longer edge times | Compare matched completed requests for the same endpoint and window. Check origin-side timing | +| **Cache miss increasing latency** | Cache hit rate dropped for affected paths. The 499 rate tracks the hit-rate change | Check cache configuration for affected paths | +| **Possible edge or network issue** | Rate notably elevated in one data center relative to matched cohorts elsewhere, with no explanation established from client timing, origin-side evidence, or cache behavior | Collect request-linked evidence and matched-cohort comparisons. Contact Cloudflare Support if unresolved | ### How to investigate -Use [Instant Logs](/logs/instant-logs/) or a [Logpush](/logs/logpush/) export filtered to `EdgeResponseStatus = 499`. No single field identifies the cause — compare 499 log entries against completed requests for the same endpoint, client population, and time window: +Use [Instant Logs](/logs/instant-logs/) or a [Logpush](/logs/logpush/) export to collect requests for the affected endpoint, client population, protocol, and time window. Keep both the `499` log entries and the other eligible requests. An export filtered only to `EdgeResponseStatus = 499` cannot provide comparison requests or the denominator needed to calculate a `499` rate. + +Calculate the `499` rate as requests with `EdgeResponseStatus = 499` divided by all eligible requests in the same population. If the logs are sampled, account for sampling consistently in both counts. For Instant Logs, use the returned `sampleInterval` values when estimating request counts. Use matched completed requests separately to compare timing fields. -- **`OriginIP`** — indicates whether an origin IP was recorded for the request. An absent value does not establish where the request stopped: cache-served responses do not require an origin fetch; an aborted or partially logged request may also lack an origin IP. Use `CacheCacheStatus` to distinguish these cases. -- **`CacheCacheStatus`** — the log field name for cache disposition in the [HTTP requests dataset](/logs/logpush/logpush-job/datasets/zone/http_requests/). If `OriginIP` is absent and `CacheCacheStatus` is HIT or STALE, the request was served from cache — the missing origin IP is expected. If `OriginIP` is absent and `CacheCacheStatus` is MISS or BYPASS, the cache phase indicates an origin fetch was expected; however, an aborted request may lack phase fields. Treat this combination as an unresolved phase and collect request-linked and origin-side evidence before attributing a cause. +No single field identifies the cause. Correlate these fields with matched request populations and request-linked evidence: + +- **`OriginIP`** — the origin IP recorded for the request. An absent value does not establish where the request stopped. A request served from cache may not require an origin fetch. An aborted or incompletely logged request may also lack an origin IP. Correlate this field with `CacheCacheStatus` and request-linked evidence. +- **`CacheCacheStatus`** — records cache disposition in the [HTTP requests dataset](/logs/logpush/logpush-job/datasets/zone/http_requests/). Values such as `hit` or `stale` indicate a cache-served response, where an absent origin IP may be expected. The `dynamic` value means the request is not eligible for cache and can include firewall-blocked requests. Values such as `miss` or `bypass` help identify requests to investigate. They do not establish that an origin connection was attempted or where an aborted request stopped. Collect request-linked and origin-side evidence before attributing a cause. - **`EdgeColoCode`** — if 499 errors are elevated at one data center, compare the 499 rate relative to eligible traffic for the same endpoint, client population, and protocol across data centers in the same window. A rate notably higher in one location than in matched cohorts elsewhere is a useful lead, but location alone does not identify an infrastructure fault. -- **UTC time window and volume trend** — compare 499 counts against completed requests in the same window and for the same endpoints. Differences in traffic mix across cohorts can produce different 499 rates without identifying a fault. +- **UTC time window and volume trend** — compare `499` rates for the same endpoint, client population, and protocol across time windows. Use all eligible requests in each matched population as the denominator. Differences in traffic mix can produce different `499` rates without identifying a fault. To investigate a 499 spike: -1. Identify which endpoints and time windows are affected. Compare 499 rates against completed requests for the same endpoint, client population, and protocol to establish a consistent baseline. -2. For the affected endpoints, check matched completed requests (not the 499 entries themselves) for elevated `OriginResponseDurationMs` values. The `OriginResponseDurationMs` field reflects the upstream response time recorded when a response completes, including routing, tiering, connection, and response time; it is not recorded in the same way for an aborted request. Use same-endpoint, same-window completed requests and origin-side logs to identify where latency is highest. -3. Check `OriginIP` and `CacheCacheStatus` for the 499 log entries and group by `EdgeColoCode`. If `OriginIP` is absent and `CacheCacheStatus` is MISS or BYPASS with no cache or origin explanation, and the rate is notably higher in one data center than in matched cohorts elsewhere in the same window, contact Cloudflare Support with the data center name, the matched-cohort 499 rate comparison, and the affected time window. + + +1. Identify the affected endpoints and time windows. Calculate the `499` rate using all eligible requests for the same endpoint, client population, and protocol. Compare that rate with the normal baseline for the same population. Use matched completed requests for the separate timing comparison in the next step. +2. Compare `OriginResponseDurationMs` on matched completed requests with the normal baseline for the same endpoint, client population, and protocol. The field measures upstream response time from the first Cloudflare data center that receives the request. It includes Argo Smart Routing and Tiered Cache time, plus time to connect to and receive a response from the origin. An elevated value does not identify where the delay occurred. Correlate it with origin-side timing and request-linked evidence. Do not use a timing value from a `499` entry alone to infer the failed phase or assign the delay to the origin server. +3. Review `OriginIP` and `CacheCacheStatus` for the affected requests. Compare the `499` rate across data centers using matched endpoint, client, and protocol populations in the same window. An absent origin IP with `miss` or `bypass` is a lead to correlate with request-linked and origin-side evidence. If the elevated rate remains unexplained, contact Cloudflare Support with the affected window, sample Ray IDs, and matched-cohort comparisons. + + ### TCP connection timing -Cloudflare's [connection limits reference](/fundamentals/reference/connection-limits/) publishes a 19-second complete TCP connection limit for the origin-connect phase. A client that disconnects during origin connection establishment will produce a 499 log entry. +The [Connection limits reference](/fundamentals/reference/connection-limits/) publishes a 19-second TCP connection establishment limit for the origin-connect phase. A client that disconnects during origin connection establishment will produce a 499 log entry. This is one phase in which a 499 can occur. Clients can also disconnect during the HTTP request phase, while response headers are being sent, or while the response body is being delivered. Identifying which phase was in progress when the client disconnected requires comparing `EdgeTimeToFirstByteMs`, `EdgeResponseBodyBytes`, and matched completed requests rather than inferring it from the 499 entry alone. ### Long-running and streaming responses -A 499 can occur before the first byte is delivered or after response-body delivery has begun, if the client disconnects before the response is complete. For example, a client with a short deadline may disconnect during origin connection or response; a client with a longer deadline may disconnect partway through receiving a large response body. +A 499 can occur before the first byte is delivered or after response-body delivery has begun, if the client disconnects before the response is complete. For example, a client with a short deadline may disconnect during origin connection or response. A client with a longer deadline may disconnect partway through receiving a large response body. For streaming responses (HLS, DASH, or other media formats) and other long-running requests, clients use varying connection and read timeout defaults, some configurable at the application level and some not. To investigate 499 errors for long-running or streaming responses: + + 1. Confirm the effective timeout in use — check the client or SDK documentation and, if possible, confirm with your development team whether the timeout is configurable. -2. In Cloudflare Logs, check `EdgeTimeToFirstByteMs` and compare it against `EdgeResponseBodyBytes` and total response duration for the 499 log entries versus completed requests on the same paths. This identifies whether the disconnect occurred at first byte or during body delivery. -3. If the affected paths are cacheable and the cache hit rate has dropped for those paths, check the cache configuration. A lower hit rate increases origin fetches and can raise latency for both first-byte and body delivery. 4. Group the 499 entries by `EdgeColoCode` and compare the rate relative to eligible traffic for the same paths and client population across data centers. If one data center is significantly elevated with no origin-side or cache explanation, contact Cloudflare Support. +2. Check `EdgeTimeToFirstByteMs` and `EdgeResponseBodyBytes` for the `499` log entries. Compare them with matched completed requests on the same paths. Positive response-body bytes support that body delivery had begun. Zero or missing body bytes do not, by themselves, establish that the disconnect occurred before the first byte. Use client-side timing or request-linked evidence when the phase remains unclear. +3. If the affected paths are cacheable and their cache hit rate has dropped, check the cache configuration. A lower hit rate increases origin fetches and can raise latency for both first-byte and body delivery. +4. Compare the `499` rate across data centers using matched paths, client populations, and protocols. Use all eligible requests in each population as the denominator. If one data center is significantly elevated with no origin-side or cache explanation, contact Cloudflare Support. + + ### Transaction-sensitive requests -When a 499 occurs on a request that was intended to modify server state, the outcome is ambiguous: the connection closed before a response was received, and the log entry alone does not establish whether the operation committed at the origin before the connection closed. +When a `499` is logged for a request intended to modify server state, the log entry alone does not establish the operation's outcome. The client may have received response headers or part of the response body before disconnecting. If the application has not confirmed the result, reconcile the operation before retrying it. -To handle ambiguous outcomes: +To handle an unconfirmed outcome: -- Reconcile the affected request and operation identifiers against your application's transaction records to determine whether the operation was recorded before the connection closed. -- Implement idempotency keys for transaction-sensitive endpoints. An idempotency key allows you to resubmit an ambiguous request and receive the result of the original operation if it was already processed, without committing it again. +- Match the affected request and operation identifiers to your application's transaction records or operation-status endpoint. Follow the application's reconciliation and retry policy if the outcome remains pending or unknown. +- Where your application supports idempotency keys, use them for state-changing endpoints. Reuse the same key when retrying the same operation, following the application's documented key scope, retention period, and retry behavior. :::note -[RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2) defines `DELETE` as idempotent in its intended effect — repeating the operation is defined to produce the same end state. This does not resolve whether the first operation was recorded before the connection closed. Reconcile against application transaction records in the same way as other state-changing requests. +[RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2) defines `DELETE` as idempotent in its intended effect. Repeating the same request is defined to have the same intended effect as making it once. This does not establish the first request's outcome. Use application-level evidence to resolve an unconfirmed result. ::: ### When to contact Cloudflare Support @@ -76,7 +91,7 @@ To handle ambiguous outcomes: Contact support when: - 499 errors spike clearly above your zone's normal baseline for the same endpoints and window, and the spike is not explained by a corresponding change in origin response time or cache hit rate. -- `OriginIP` is absent and `CacheCacheStatus` is MISS, BYPASS, or DYNAMIC for the affected log entries, with no cache or origin-side explanation, and matched cohort comparison across data centers does not isolate the cause. +- The affected requests have no recorded `OriginIP` and a `CacheCacheStatus` of `miss` or `bypass`, and the elevated `499` rate remains unexplained after reviewing matched request populations, client timing, and origin-side evidence. Include these field values as investigation clues, not as proof of a Cloudflare fault. - The 499 rate is significantly elevated in one data center relative to matched traffic in other data centers serving the same zone in the same window, with no origin-side or cache explanation. When you contact support, include: @@ -87,4 +102,4 @@ When you contact support, include: - The `OriginIP` and `CacheCacheStatus` values for those log entries - The `EdgeColoCode` distribution for the 499 entries and the same-window 499 rate in matched cohorts at other data centers - Whether other zones or hostnames experienced the same spike simultaneously -- For Enterprise customers: a [Logpush](/logs/logpush/) export filtered to `EdgeResponseStatus = 499` for the time window +- For Enterprise customers: a [Logpush](/logs/logpush/) export containing the affected `499` entries and matched comparison requests for the time window diff --git a/src/content/docs/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-522.mdx b/src/content/docs/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-522.mdx index acb86386ecf..229aedb50b2 100644 --- a/src/content/docs/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-522.mdx +++ b/src/content/docs/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-522.mdx @@ -9,24 +9,37 @@ products: ## Error 522: connection timed out -Error `522` occurs when Cloudflare times out contacting the origin web server. +Error `522` means Cloudflare reported a connection timeout while trying to reach the website's origin server. -### Common causes +### Connection setup and reuse -Two different timeouts cause HTTP error `522` depending on when they occur between Cloudflare and the origin web server: +For a new TCP-based HTTPS connection, setup includes the TCP handshake followed by SSL/TLS negotiation. TCP can already be established while HTTPS setup remains unfinished. A timeout during that unfinished setup can be reported as `522`. A `522` therefore does not necessarily mean the TCP handshake failed. -- Before a TCP connection is established, Cloudflare does not receive a SYN+ACK within 19 seconds after sending a SYN. The SYN retry backoff intervals are 1, 1, 1, 1, 1, 2, 4, and 8 seconds. -- After the TCP connection is established, Cloudflare does not receive an acknowledgment (ACK) of its resource request within 90 seconds. +Requests can also reuse established connections. A new TCP/TLS handshake is not required for every request. + +One documented `522` condition is a timeout waiting for TCP acknowledgment of a request on an established connection. A TCP acknowledgment is distinct from an HTTP response. This means `522` is not limited to initial connection setup. + +`522` reports a connection timeout. [525](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-525/) reports an SSL/TLS handshake failure. These codes do not simply separate TCP errors from TLS errors. A timeout while TLS is pending can be reported as `522` rather than `525`. The error code alone does not identify the exact timed operation or what caused this request's failure. + +### Documented TCP timeout conditions + +The [Connection limits reference](/fundamentals/reference/connection-limits/) documents these TCP timeout conditions. They are not an exhaustive list of every operation that can produce `522`: + +- **Before TCP establishment:** Cloudflare does not receive a SYN+ACK within 19 seconds after sending a SYN. The documented SYN retry backoff intervals are 1, 1, 1, 1, 1, 2, 4, and 8 seconds. +- **After TCP establishment:** Cloudflare does not receive a TCP acknowledgment of its resource request within 90 seconds. + +These values apply to the named TCP conditions. Do not interpret either number as a universal TCP-plus-TLS setup budget or as the measured duration of this request's failure. ### Resolution -- Contact your hosting provider and share the [error details](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/#required-error-details-for-hosting-provider) to assist in troubleshooting these common causes: +- Review firewall, security, rate-limiting, and network settings for your connection path first. Blocked or rate-limited Cloudflare IP ranges at the origin or an intermediate firewall are the most common cause of `522` errors. For standard proxied origin connections, confirm that your rules allow [all published Cloudflare IP ranges](https://www.cloudflare.com/ips/). For [Dedicated CDN Egress IPs](/smart-shield/configuration/dedicated-egress-ips/), check the addresses assigned to your setup. +- Verify the configured origin address and port. Confirm the service is running and accepting connections. Review its capacity and logs for overload or connection limits, and review its [HTTP keep-alive settings](/fundamentals/reference/connection-limits/#keep-alives). +- Review origin and network logs for the reported time. If the origin connection uses HTTPS, include SSL/TLS negotiation in the investigation. +- If you use Cloudflare Tunnel, review the connector and local service logs using the [Tunnel troubleshooting guide](/tunnel/troubleshooting/). +- Ask your hosting provider or network administrator to investigate the connection timeout and relevant connection-path settings. Share the [error details](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/#required-error-details-for-hosting-provider). +- If the problem persists, contact Cloudflare Support with the Ray ID, affected URL, timestamp and time zone, and relevant findings. - - [Cloudflare IP addresses](https://www.cloudflare.com/ips/) are rate limited or blocked in .htaccess, iptables, or firewalls. Confirm your hosting provider allows **all Cloudflare IP ranges** (most common cause). You can use a [Cloudflare WAF Custom Rule](/waf/custom-rules/use-cases/block-by-geographical-location/) if you need to restrict traffic from geographical locations. - - An overloaded or offline origin web server drops incoming requests. - - [Keepalives](http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/overview.html) are disabled at the origin web server. - - The origin IP address in your Cloudflare **DNS** app does not match the IP address currently provisioned to your origin web server by your hosting provider. - - Packets were dropped at your origin web server. +### Product-specific checks - If you are using [Cloudflare Pages](/pages/), verify that you have a custom domain set up and that your CNAME record is pointed to your [custom Pages domain](/pages/configuration/custom-domains/#add-a-custom-domain). @@ -34,10 +47,12 @@ Two different timeouts cause HTTP error `522` depending on when they occur betwe - If you are using [Origin Rules](/rules/origin-rules/), make sure the resulting hostname can be resolved. For example, if your Origin Rule point to a Worker route, a `522` error will be returned if the hostname for this route is an A record pointing to a reserved address such as `100::` or `192.0.2.0`. -- If none of the above leads to a resolution, request the following information from your hosting provider or site administrator before [contacting Cloudflare support](/support/contacting-cloudflare-support/): +### Information for Cloudflare Support + +Request the following information from your hosting provider or site administrator before [contacting Cloudflare Support](/support/contacting-cloudflare-support/): - - An [MTR or traceroute](/support/troubleshooting/general-troubleshooting/gathering-information-for-troubleshooting-sites/#perform-a-traceroute) from your origin web server to a [Cloudflare IP address](http://www.cloudflare.com/ips) that most commonly connected to your origin web server before the issue occurred. Identify a connecting Cloudflare IP recorded in the origin web server logs. - - Details from the hosting provider's investigation, such as pertinent logs or conversations with the hosting provider. +- An [MTR or traceroute](/support/troubleshooting/general-troubleshooting/gathering-information-for-troubleshooting-sites/#perform-a-traceroute) from your origin web server to a [Cloudflare IP address](https://www.cloudflare.com/ips/) that connected to your origin web server before the issue occurred. Identify the connecting address in origin logs. This probes the route from the origin to that address. It does not, by itself, establish the forward route used for the failed request. +- Details from the hosting provider's investigation, such as pertinent logs or conversations with the hosting provider. ### Diagnose with Origin Analytics diff --git a/src/content/docs/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-525.mdx b/src/content/docs/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-525.mdx index 663961a09a6..a84d47efaa6 100644 --- a/src/content/docs/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-525.mdx +++ b/src/content/docs/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-525.mdx @@ -11,7 +11,13 @@ import { GlossaryTooltip } from "~/components"; ## Error 525: SSL handshake failed -This error indicates that the SSL handshake between Cloudflare and the origin web server failed. +Error `525` means Cloudflare reported an SSL/TLS handshake failure between Cloudflare and the origin web server. + +### How this differs from Error 522 + +[522](/support/troubleshooting/http-status-codes/cloudflare-5xx-errors/error-522/) reports a connection timeout. `525` reports an SSL/TLS handshake failure. These codes do not simply separate TCP errors from TLS errors. A timeout while TLS is pending can be reported as `522` rather than `525`. + +For a new TCP-based HTTPS connection, TCP establishment is followed by SSL/TLS negotiation. TCP can already be established while HTTPS setup remains unfinished. Refer to [Connection limits](/fundamentals/reference/connection-limits/#what-connection-means-here) for the qualified terms and documented TCP timeout conditions. ### Common causes