Repository navigation
SE.Redis Connection Storms & High CPU #3036
Description
Description
When StackExchange.Redis connects to a Redis instance through a proxy layer (e.g., Envoy, HAProxy, or cloud-managed Redis proxies like Azure Managed Redis), a backend node failure triggers an infinite, high-frequency connection storm that bypasses the library's default exponential backoff algorithm.
This results in the client forcefully DDoSing the proxy with hundreds of connection attempts per second per instance, leading to SNAT port exhaustion, ThreadPool starvation, and proxy-layer CPU spikes.
To Reproduce
The issue stems from how cloud proxies handle load-shedding when their backend shards are dead or unresponsive (e.g., hitting 100% CPU).
- Client (
SE.Redis) sits behind a TCP/TLS Proxy. The Proxy sits in front of the Redis server. - The backend Redis server becomes completely unresponsive.
- The Proxy process is still healthy, so it accepts the client's incoming TCP handshake (
SYN->SYN-ACK). SE.Redistreats this as a successful connection establishment and resets its internal reconnect backoff counters back to zero.- The Proxy attempts to route the connection/initial commands to the backend Redis node, realizes the node is dead, and violently sheds the load by dropping the socket (
SO_LINGER = 0). - The client OS receives a
TCP RST(surfaced asSocketException 10054: An existing connection was forcibly closed by the remote host). - Because the failure counter was just reset to zero (due to the successful TCP proxy handshake),
SE.Redisimmediately attempts to reconnect without any delay. - The loop repeats infinitely.
Expected behavior
The ConnectionMultiplexer should recognize that the connection was dropped during the initial routing/handshake phase (before actual Redis commands or heartbeats successfully flow) and should correctly apply the default Reconnect Retry Policy (exponential backoff).
Since we did not explicitly configure a custom ReconnectRetryPolicy in our ConfigurationOptions, we expected the default backoff to prevent a storm. Is it possible for the library to not treat a mere TCP SYN-ACK from a proxy as a fully healthy connection that warrants resetting the consecutive failure penalty?
Actual behavior
The library enters a tight, infinite loop. Our proxy-level telemetry showed operations/sec spiking from a baseline of ~50k to over 100,000+ operations/sec consisting entirely of failed connection handshakes, while actual GET/SET commands were flat at zero.
Environment details
- StackExchange.Redis Version: v2.11.8
- OS: Windows / Linux
- .NET Version: .NET 8.0
- Architecture: Distributed clients connecting via an intermediate proxy to a clustered Redis backend.
Additional context
We verified this behavior locally using a simple Python TCP relay to mimic the proxy. If the relay accepts the connection but forcefully closes it (l_linger = 0) before routing data, StackExchange.Redis falls into the exact same connection storm loop.
We also confirmed that if we explicitly inject a custom IReconnectRetryPolicy into the configuration options that forces a delay or rejects retries, the library does obey it and mitigates the storm. This isolates the issue to how the default state machine handles the reset counter upon receiving a proxy's SYN-ACK followed by an immediate connection drop.
Activity
mgravell commentedon Mar 13, 2026
Huh. Totally agree, we shouldn't reset any clocks unless we successfully complete a handshake through to the "critical tracer" point (an arbitrary
ECHO/PINGthat we issue after TLS,AUTH/HELLO/etc, that is used to confirm that we have established a conversation).Will try to add to our backlog, hopefully for ASAP.
Pavanlalit commentedon Mar 13, 2026
Awesome, thanks @mgravell . That completely makes sense. Waiting for the ECHO/PING to succeed before resetting the failure clocks will perfectly solve the proxy load-shedding trap we're seeing in production.
(Just as a quick extra data point: we did confirm locally that if we explicitly inject a custom IReconnectRetryPolicy that forces a delay, the library DOES track the counter and throttle correctly. The issue is strictly isolated to the default policy/clock getting reset by the proxy's TCP handshake before the critical tracer completes).'
Really appreciate the fast response and you looking into it! Let me know if you need any testing or traces from our end when a fix is ready.
Pavanlalit commentedon Mar 14, 2026
Awesome, thanks Marc. That completely makes sense. Really appreciate you looking into it..
While we wait for that patch to make its way through your backlog and release cycle, we need to deploy a temporary stopgap to protect our proxies from the 1M+ Ops/sec SYN storm. We are weighing our options and would love your recommendation:
** A Custom IReconnectRetryPolicy:**
We prototyped a custom policy that ignores the currentRetryCount (since the proxy bug keeps resetting it to 0) and simply enforces a hard minimum floor delay with jitter to safely stagger the cluster's recovery. Caveat: I noticed in a previous discussion (#2557) @NickCraver mentioned that policies can be ill-suited for managing these types of reconnect loops, so I want to make sure we aren't fighting the library.
The ForceReconnect Pattern
Alternatively, we could implement the ForceReconnect pattern outlined in the Azure Managed Redis Best Practices. Suggesting to catch the continuous SocketExceptions at the application level and physically .Dispose() and recreate the ConnectionMultiplexer to forcefully kill the background thread.
Another approach?
If neither of the above is ideal, is there a better, recommended way to physically throttle or pause the multiplexer during these proxy load events until the tracer fix is live?
Thanks again for your time and the quick validation on the bug!
DrEsteban commentedon Mar 14, 2026
Just to note: This issue particularly affects the Redis Enterprise stack from Redis Inc, since they have a custom "DMC Proxy" sitting in front of a set of Redis shards on the host.
In a failover scenario, all clients will mass-migrate from one host to another. This proxy layer (masquerading as a single Redis shard) will "answer the phone", but may be unable to complete the Redis handshake in time due to the onslaught of incoming connections. This should settle out eventually if clients follow proper backoff retries, but with thousands of clients stuck in this tight retry loop it has the potential to DDoS the cache.
Pavanlalit commentedon Mar 14, 2026
Thanks forntheninsights @DrEsteban ..
Ours is a high traffic service whcih handles billions of reqs per day and keeping redisnhealthy is a crucial part for us.
So jittered backoff retries wouldnt help here for our case?
If so, what are our alternatives until we get the se.redis library to get the fix out?
tests for args passed to retry-policy
mgravell commentedon Mar 19, 2026
See #3038 for investigation/resolution; I will get this (and #3037, which also touches on handshake scenarios) released ASAP.
Improve detection of connect/handshake failures and the retry-policy (#…
mgravell commentedon Mar 19, 2026
https://github.com/StackExchange/StackExchange.Redis/releases/tag/2.12.4