Repository navigation
Endless connection failures to old nodes after Azure Managed Redis configuration change #3054
Description
Summary
We have an application that uses StackExchange.Redis to connect to Azure Managed Redis.
During and following a maintenance event, we observed repeated connection failures being logged. These continued even after the maintenance event had completed, apparently indefinitely.
The application did not appear to be significantly affected in terms of functionality - Redis commands were still being executed successfully. My investigation suggests that the failed connections were to nodes that are no longer part of the cluster.
Steps to reproduce
I can reproduce this using a simple command-line program (listing below).
I run this, connecting to a small Azure Managed Redis instance.
Once it has connected, I go to the AMR's Scale settings in the Azure Portal and change the target cache size from 0.5 GB to 1.0 GB (or vice-versa).
As the change is happening, the program starts logging UnableToConnect errors similar to:
UnableToConnect on [redacted IP]:8501/Subscription, Initializing/NotStarted, last: SUBSCRIBE, origin: BeginConnectAsync, outstanding: 0, last-read: 2s ago, last-write: 2s ago, keep-alive: 60s, state: Connecting, mgr: 10 of 10 available, last-heartbeat: never, last-mbeat: 0s ago, global: 0s ago, v: 2.12.14.58429
These continue indefinitely.
However, restarting the application connects successfully and does not log these errors.
Program listing:
using Azure.Identity;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using StackExchange.Redis;
var appBuilder = Host.CreateApplicationBuilder();
appBuilder.Configuration.AddCommandLine(args);
appBuilder.Logging.AddSeq();
var app = appBuilder.Build();
const string settingName = "redis";
string connectionString = appBuilder.Configuration[settingName] ?? throw new InvalidOperationException($"Specify Redis connection string in setting {settingName}");
var configurationOptions = ConfigurationOptions.Parse(connectionString);
await configurationOptions.ConfigureForAzureWithTokenCredentialAsync(new DefaultAzureCredential());
configurationOptions.LoggerFactory = app.Services.GetRequiredService<ILoggerFactory>();
var connectionMultiplexer = await ConnectionMultiplexer.ConnectAsync(configurationOptions);
Console.ReadKey();
GC.KeepAlive(connectionMultiplexer);Investigation
"Server summary" logging from before the change:
Server summary: [redacted hostname]:10000: Cluster v7.4.3, primary; keep-alive: 00:01:00; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active [...]
Server summary: [redacted IP]:8500: Cluster v7.4.3, primary; keep-alive: 00:01:00; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active [...]
Server summary: [redacted IP]:8501: Cluster v7.4.3, primary; keep-alive: 00:01:00; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active [...]
And the same after the change:
Server summary: [redacted hostname]:10000: Cluster v7.4.3, primary; keep-alive: 00:01:00; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active
Server summary: [redacted IP]:8500: Cluster v7.4.3, primary; keep-alive: 00:01:00; int: Connecting; sub: Disconnected [...]
Server summary: [redacted IP]:8501: Cluster v7.4.3, primary; keep-alive: 00:01:00; int: Disconnected; sub: Connecting [...]
Server summary: [redacted IP]:8503: Cluster v7.4.3, primary; keep-alive: 00:01:00; int: ConnectedEstablished; sub: ConnectedEstablished, 2 active [...]
Server summary: [redacted IP]:8502: Cluster v7.4.3, primary; keep-alive: 00:01:00; int: ConnectedEstablished; sub: ConnectedEstablished, 1 active [...]
Doing some debugging, I can see that during the change, the ServerEndPoint instances for the service endpoint [redacted hostname]:10000 and the two new nodes all receive at least one updated ClusterConfiguration with only the two new nodes in it. The ServerEndPoint instances for the existing nodes have their connections closed and never receive a new ClusterConfiguration.
After restarting the application, the "Server summary" logging once it has connected is:
Server summary: [redacted hostname]:10000: Cluster v7.4.3, primary; keep-alive: 00:01:00; int: ConnectedEstablished; sub: n/a [...]
Server summary: [redacted IP]:8502: Cluster v7.4.3, primary; keep-alive: 00:01:00; int: ConnectedEstablished; sub: n/a [...]
Server summary: [redacted IP]:8503: Cluster v7.4.3, primary; keep-alive: 00:01:00; int: ConnectedEstablished; sub: n/a [...]
So it looks like the ServerEndPoint instances for the nodes that are no longer part of the cluster just keep trying to reconnect unsuccessfully, resulting in all the failures being logged.
I did a bit of code browsing/debugging but didn't find any obvious mechanism for such nodes to be removed from the connection multiplexer.
Versions etc
Client:
- StackExchange.Redis 2.12.14
- Microsoft.Azure.StackExchangeRedis 3.3.1
- .NET 10/Windows 11
Server:
- Azure Managed Redis
- Redis 7.4
- Clustering Policy: OSSCluster
Activity
mgravell commentedon Apr 11, 2026
Interesting. Will investigate - sounds like we need to do better culling. Very useful context / research that you have included, thanks.
natelaff commentedon Apr 13, 2026
Experiencing the same pattern on Azure Managed Redis (Balanced_B0, OSSCluster policy, HA enabled, zone redundant, Entra ID auth).
Three incidents in 8 days, all identical behavior. The client discovers internal cluster node IPs via CLUSTER NODES and connects directly. When a node goes down, the multiplexer is stuck retrying the dead internal IP indefinitely — even though the proxy endpoint (:10000) reconnects successfully.
OTel traces show the server summary just before failure:
Traffic routes primarily through the internal nodes (176 sockets), not the proxy. When :8502 goes down, every consuming service throws UnableToConnect for 29 minutes straight (~200 exceptions per service). Restarting the application doesn’t help — the fresh ConnectionMultiplexer rediscovers the same dead node via CLUSTER NODES because the cluster topology still includes it.
The dead node is never removed from the cluster topology during the outage. Once Azure restores the node, errors stop immediately.
Key details:
Our planned workaround is recreating the database with EnterpriseCluster policy, which blocks CLUSTER NODES/SLOTS and forces all traffic through the proxy.
Kumima commentedon May 22, 2026
Met the same problem under nearly 100% same environment. Also using
Azure Managed Redis Balanced_B0, Redis 7.4and Aspire'sbuilder.AddKeyedRedisClientBuilder(key).WithAzureAuthentication().WithDistributedCache();That log is error level, which is causing my logging database to grow continuously.
ibauersachs commentedon Aug 5, 2026
This is likely the same as #2417. Swapping a ConnectionMultiplexer is painful, as it's typically in the DI container. A replica no longer reported by
SENTINEL REPLICASorCLUSTER REPLICASreally should be removed from the connection list.mgravell commentedon Aug 24, 2026
The cause looks structural rather than incidental: nothing in the library could ever remove a
ServerEndPoint.ServerSnapshothad anAddand no counterpart, nothing calledservers.Remove, andDisposewas only reached when the whole multiplexer went away. So a node that left the cluster stayed in the server list and kept being reconnected to, indefinitely - which matches the symptom here of failures continuing long after the scale operation finished, while commands keep working (routing had already moved on; the dead entries were just noise).#3177 adds the missing half: endpoints are tracked against the topology that reported them, and one that stops being listed is drained and removed once it is idle - owns no slots, carries no subscriptions, owes no responses. Explicitly configured endpoints are never removed, since they are the seeds needed to reconnect after everything else has rotated out.
Two caveats before treating this as fixed:
marc/cluster-slots-routing), that would be genuinely useful.CLUSTER SLOTSreply reflects one node's view and can be briefly stale. So expect a short tail of failures after a scale event rather than an immediate stop.natelaff commentedon Aug 24, 2026
I did migrate to enterpriseCluster which sort of helped but not really. the biggest issue is these azure linux instances (wasn't an issue on the windows app services) and how that TCP holds. at least i have it in a place now when one of these situations occurs i have about a 10 second blip rather than a 90+ minute incident. but i will try to get this weaved in shortly. thank you for the update!
mgravell commentedon Aug 24, 2026
@philon-msft this change is not GA yet, but in advance: any tips on validating this scenario, i.e. setting up an AMR that cycles nodes out, so we can check they aren't "clingy"?
chris-white-wtw commentedon Sep 2, 2026
Hi @mgravell, sorry for not responding earlier - I've just returned from leave.
I've just tried the 3.1.37 pre-release and compared it with 3.1.31, connecting to an AMR which I then re-scale to provoke the cluster reconfiguration. I can see that whereas 3.1.31 continues to log connection failures as in the original report, 3.1.37 prunes the old nodes after around 30 seconds and no further errors are logged after that, which I believe matches the behaviour you're expecting with the fix.
vanwx commentedon Sep 4, 2026
Hi @chris-white-wtw how can I get the pre-release version 3.1.37 pre-release
mgravell commentedon Sep 4, 2026
I assume they mean they built from source. I plan to release in the next day or so, which will be the next actual release.
vanwx commentedon Sep 4, 2026
Thanks @mgravell
chris-white-wtw commentedon Sep 4, 2026
Hi @vanwx, @mgravell - I actually used the MyGet feed (https://www.myget.org/feed/stackoverflow/package/nuget/StackExchange.Redis)