Skip to content

TCP keep-alive failing on DNS endpoints pre .NET 10 #3086

Closed

Description

@liukonen

using VALKEY

Unhandled exception. StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code.

at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl(ConfigurationOptions configuration, TextWriter writer, Nullable`1 serverType, EndPointCollection endpoints) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 691

at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 663

at StackExchange.Redis.ConnectionMultiplexer.Connect(String configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 641

at StocksRssFeed.Startup.ConfigureServices(IServiceCollection services)

at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)

at System.Reflection.MethodBaseInvoker.InvokeDirectByRefWithFewArgs(Object obj, Span`1 copyOfArgs, BindingFlags invokeAttr)

at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)

at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services, Object instance)

Activity

  1. mgravell commented on May 22, 2026

    @mgravell
    Collaborator

    Well that's definitely not intentional! I've marked that build as hidden for now; will investigate.

  2. mgravell commented on May 22, 2026

    @mgravell
    Collaborator

    Question: what version are you upgrading from? I'm trying to but bookends on where I look for deltas.

  3. mgravell commented on May 22, 2026

    @mgravell
    Collaborator

    OK, I've had time to look, and I'm not seeing any fundamental connection problems with Valkey or any other backend - for example, testing locally against valkey_version:9.1.0: it works fine.

    So right now I have no clue; I'm going to need your help!

    Does the library fail for you when targeting a local Valkey server? for example docker container run --name valkey -p 6379:6379 -d valkey/valkey? (this spins up the current default valkey dockerhub image locally and exposes it to the host OS on port 6379). If that then fails to connect, it is probably something like assembly binding - we'd need to figure out what.

    After that there's a few ways we can look into this:

    1. pass a StringWriter (or similar) into the log parameter of Connect, so we can get verbose information about every step in connect and handshake; note that this may include server names etc, so you might want to eyeball it for anything privileged
    2. you could enable RESP logging; this WILL contain any AUTH etc tokens, but provides the absolute most detail for any conversation between the endpoints

    I've masked the release while I investigate, but: "works for meTM", so I'm going to need some help to get to the bottom of this.

    Obviously don't post full logs here; I am available at marc.gravell@gmail.com and marc.gravell@redis.com

  4. liukonen commented on May 23, 2026

    @liukonen
    Author

    Hi Marc,

    Thanks — I ran the local Valkey repro exactly as suggested:

    docker run --name valkey -p 6379:6379 -d valkey/valkey
    Result
    ✔ Valkey is reachable on localhost:6379
    ✔ Connection works locally against the same container
    ✔ Same StackExchange.Redis version that fails in our Docker network scenario
    ✔ No failure reproduced in this local single-container setup

    So:
    local host repro works fine
    Key difference vs failing environment
    The failure only occurs when:

    • App runs in Docker container
    • Valkey runs in separate container
    • Connected via Docker bridge network (service name DNS: valkey)

    In that setup, latest StackExchange.Redis:

    • Resolves endpoint ✔
    • Begins connect ✔
    • Waits for response ❌
    • Times out after ~10s
    • Repeats reconfiguration cycles
    • Never completes handshake

    Previous StackExchange.Redis version:

    Same environment
    Same network
    Connects successfully immediately
    Logging (as requested)

    I captured verbose connect logs via StringWriter / Console.Out.

    Key pattern:

    BeginConnectAsync
    Allowing 1 endpoint(s) 00:00:10 to respond...
    Did not respond
    Timeout 10000ms
    WaitingForActivation
    No primaries detected

    No:

    AUTH errors
    TLS errors
    protocol negotiation failures

    Just silent non-response during handshake window.

    RESP logging

    I can enable RESP-level logging if needed, but current logs already show that the failure occurs before any meaningful protocol exchange completes.

    Conclusion
    Local single-container Valkey: ✔ works
    Multi-container Docker network: ❌ fails (latest client only)
    Prior StackExchange.Redis version: ✔ works in same multi-container setup

    So the regression appears specific to container-to-container networking behavior in the latest client, not Valkey compatibility in general or single-node connectivity.

    Attached the output of my docker logs. I dont see anything privileged

    Also, I am running this in DotNet 8 on x64 hardware using ubuntu 24.04.4, and using microsoft Chisled repos as my base image

    _stocksrssfeed_logs.txt

    My "internal" network, where my containers live, is set up as

        "Name": "interlink",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv4": true,
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
        },
        "ConfigFrom": {
            "Network": ""
        },
    
  5. mgravell commented on May 23, 2026

    @mgravell
    Collaborator

    Ok, that's super interesting, thanks. Will look after the weekend (UK public holiday). If you have any specific docker config for that setup you can share, that might save some time and remove any false paths, but if not: I'll do what I can.

  6. wipiano commented on May 23, 2026

    @wipiano
  7. wipiano commented on May 23, 2026

    @wipiano
  8. liukonen commented on May 23, 2026

    @liukonen
    Author

    Reproducible console app
    If I migrate to dotnet 10, the code works, so it seems to be isolated to dotnet 8
    https://github.com/liukonen/SR_Bug

  9. wipiano commented on May 23, 2026

    @wipiano
    Contributor

    After more careful testing I realized I had switched my base image from mcr.microsoft.com/dotnet/runtime:8.0-noble-chiseled to mcr.microsoft.com/dotnet/runtime:10.0 (non-chiseled) at some point during my investigation.
    The "I can't reproduce anymore" observations was made on the non-chiseled .NET 10 image.

    Going back to 8.0-noble-chiseled, the failure reproduces reliably, and crucially the SE.Redis version does matter.

    So this is a 2.13.10 regression, but it only surfaces on chiseled .NET 8 base images.

    Sorry for the noise - I should have pinned the image tag from the start.

  10. wipiano commented on May 23, 2026

    @wipiano
    Contributor

    I think the cause is in the .NET runtime layer, not StackExchange.Redis directly, but #3078 is the trigger because it added the SetSocketOption(SO_KEEPALIVE, ...) call.

    On the .NET 8.0 chiseled image, calling Socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, [true/false]) on a socket constructed via the 2-arg new Socket(SocketType, ProtocolType) ctor before calling Socket.ConnectAsync(SAEA) with a DnsEndPoint causes Socket.ConnectAsync to immediately throw:

    PlatformNotSupportedException: "Sockets on this platform are invalid for use after a failed connection attempt."

    minimal PoC without StackExchange.Redis:

    using System.Net;
    using System.Net.Sockets;
    
    using var stdout = Console.OpenStandardOutput();
    using var sw = new StreamWriter(stdout);
    sw.AutoFlush = true;
    
    await Task.Delay(500); // wait valkey server start
    
    await TestAsync(true, sw);
    await TestAsync(false, sw);
    await TestAsync(null, sw);
    
    async Task TestAsync(bool? keepAlive, StreamWriter sw)
    {
        using var socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
        if (keepAlive.HasValue)
        {
            socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.KeepAlive, keepAlive.Value);
            sw.WriteLine("SO_KEEPALIVE={keepAlive}");
        }
        else
        {
            sw.WriteLine("No SetSocketOption call");
        }
    
        var tcs = new TaskCompletionSource();
        var args = new SocketAsyncEventArgs
        {
            RemoteEndPoint = new DnsEndPoint("valkey", 6379),
        };
        args.Completed += (_, _) => tcs.TrySetResult();
    
        bool pending;
        try
        {
            pending = socket.ConnectAsync(args);
        }
        catch (Exception ex)
        {
            Console.WriteLine($"ConnectAsync threw: {ex.GetType().Name}: {ex.Message}");
            return;
        }
    
        if (pending)
        {
            var done = await Task.WhenAny(tcs.Task, Task.Delay(TimeSpan.FromSeconds(10)));
            if (done != tcs.Task)
            {
                Console.WriteLine("TIMEOUT");
                return;
            }
        }
        
        Console.WriteLine($"SocketError={args.SocketError}");
    }
    
    await sw.FlushAsync();

    Confirmed result matrix (chiseled .NET 8 / valkey on same internal bridge):

    Pre-connect call socket.ConnectAsync(SAEA{ DnsEndPoint})
    SetSocketOption(KeepAlive, true) throws PlatformNotSupportedException
    SetSocketOption(KeepAlive, false) throws PlatformNotSupportedException
    no SetSocketOption call succeeds (SocketError.Success)

    I only tested SO_KEEPALIVE, not other socket options - the trigger may be narrower or broader than that.

    Verified fix: moving the SetSocketOption(SOL_SOCKET, SO_KEEPALIVE, true) call from SocketManager.CreateSocket to after the Socket.ConnectAsync completes elimnates the PNSE and restores connectivity on chiseled .NET 8.

  11. wipiano commented on May 23, 2026

    @wipiano
    Contributor

    Confirmed with a small repro that the problematic version of StackExchange.Redis does throw the PlatformNotSupportedException mentioned above, and that it gets silently swallowed:

    AppDomain.CurrentDomain.FirstChanceException += (_, e) =>
    {
        if (e.Exception is PlatformNotSupportedException pnse)
        {
            Console.WriteLine($"[FirstChance] {pnse.GetType().Name}: {pnse.Message}");
            Console.WriteLine(pnse.StackTrace);
            Console.WriteLine("------");
        }
    };
    
    using var mul = ConnectionMultiplexer.Connect("valkey");
    Console.WriteLine(mul.GetDatabase().StringIncrement("test"));

    Output:

    consoleapp1-1  | [FirstChance] PlatformNotSupportedException: Sockets on this platform are invalid for use after a failed connection attempt.
    consoleapp1-1  |    at System.Net.Sockets.Socket.ThrowMultiConnectNotSupported()
    consoleapp1-1  | ------
    consoleapp1-1  | [FirstChance] PlatformNotSupportedException: Sockets on this platform are invalid for use after a failed connection attempt.
    consoleapp1-1  |    at System.Net.Sockets.Socket.ThrowMultiConnectNotSupported()
    consoleapp1-1  | ------
    consoleapp1-1  | Unhandled exception. StackExchange.Redis.RedisConnectionException: It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code.
    consoleapp1-1  |    at StackExchange.Redis.ConnectionMultiplexer.ConnectImpl(ConfigurationOptions configuration, TextWriter writer, Nullable`1 serverType, EndPointCollection endpoints) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 691
    consoleapp1-1  |    at StackExchange.Redis.ConnectionMultiplexer.Connect(ConfigurationOptions configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 663
    consoleapp1-1  |    at StackExchange.Redis.ConnectionMultiplexer.Connect(String configuration, TextWriter log) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 641
    consoleapp1-1  |    at Program.<Main>$(String[] args) in /src/ConsoleApp1/Program.cs:line 73
    consoleapp1-1 exited with code 139
    
  12. mgravell commented on May 23, 2026

    @mgravell
    Collaborator

    This is super, thanks. Very much appreciated. I think you can see how if I was investigating from a cold start, even finding a failing setup would have been tricky, so: thanks.

    What I propose, then:

    1. Make TCP keep-alive an explicit option in config options
    2. I'm tempted to default it to on for .NET 10+, else off
    3. If not explicitly enabled (i.e. if on by default only), I'm tempted to automatically disable it if the first connect attempt fails

    Thoughts on any of these very welcome!

  13. liukonen commented on May 23, 2026

    @liukonen
    Author

    I will be migrating my app to dotnet 10 to resolve this. That said I want to thank both of you for the quick responses on this! I know the lts support of dotnet 8 expires soon, so this is kind of a sign I need to migrate up. Can't personally speak for anyone else who might be experiencing the issue, but I'd personally probably put a line in the sand on 2.13.1 as the last supported version for dotnet 8. But that's only my 2 cents.

  14. liukonen commented on May 23, 2026

    @liukonen
    Author

    Woops accidentally clicked the close button.

  15. mgravell commented on May 23, 2026

    @mgravell
    Collaborator

    I usually aim to support 2xLTS plus final netfx

  16. wipiano commented on May 24, 2026

    @wipiano
    Contributor

    I checked the .NET 8 implementations of Socket.Unix.cs and SafeSocketHandle.Unix.cs, and the issue actually reproduces unconditionally whenever SO_KEEPALIVE is set and a DnsEndPoint is used. It does not depend on the image being chiseled. My earlier "chiseled-only" framing wasn't accurate.

    So with SE.Redis's current keep-alive implementation, any Linux user on pre-.NET 10 connecting via a DnsEndPoint will hit this. This was fixed runtime-side in dotnet/runtime#111676, which landed in release/10.0 but was not backported to release/8.0 / release/9.0.

    This maps to your option 2 ("default on for .NET 10+, else off"), but a narrower variant might be possible. To keep keep-alive enabled in as many cases as possible:

    1. On .NET 10+, behave exactly as today.
    2. On pre-.NET 10 TFMs, check the endpoint passed into SocketManager.CreateSocket(). Only call SetSocketOption(SocketOptionName.KeepAlive, ...) when endpoint is not DnsEndPoint. (Skipping the call entirely matters: passing SocketOptionName.KeepAlive is itself what sets ExposedHandleOrUntrackedConfiguration, regardless of the boolean value.)

    Happy to defer to your call.

  17. mgravell commented on May 24, 2026

    @mgravell
    Collaborator

    Thank you. Genuinely appreciate the extra investigation here; I probably would have got to the same place, but: you've saved me some work, and I appreciate that. It also explains why I didn't see it in local spot-testing: I almost certainly used anIPEndPoint. It would not have occurred to me that they would behave differently! (I did anticipate Unix sockets not wanting to act the same way, so excluded that).

  18. changed the title [-]2.13.10 suddenly giving errors[/-] [+]TCP keep-alive failing on DNS endpoints pre .NET 10[/+] on May 24, 2026
  19. mgravell commented on May 24, 2026

    @mgravell
    Collaborator

    Running tests locally - it is a weekend and the laptop available is Windows; I can confirm no-repro on Windows on all runtimes, which matches your statement above. I usually use linux; I'll re-confirm on there later, and we can presumably add another dimension to our rules matrix. Gotta love cross-platform work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions