Cluster: support host names when available, via CLUSTER SLOTS

Situation:

especially from v7, cluster shards may be routed via hosts instead of IPs
thus, the advertised IPs may sometimes (with known examples) may not be routable
SLOTS and SHARDS (v7+) provide the hosts; NODES does not, as far as known

Proposal:

prefer SLOTS to NODES (but not SHARDS due to v7 dependency)
route via hostname when available, IP otherwise
possibly with option to use fallback old behaviour?

Was: "Lookup by endpoint should check IP and host"

Currently, only exact endpoint matches are considered. However, specially in the case of CLUSTER, a node may have both host and IP evidently. Further, -MOVED may return the unexpected option, leading to additional connections.

We should:

record both identities as declared, for example from the CLUSTER NODES response
if no equality endpoint match is found, check also using the best data available (after type-testing the endpoint)

Context: dotnet/aspnetcore#59211

Auxiliary consideration: should cluster routing be limited to IP, as current? Should there be a "use host routing" option? Or should we try IP and use host if that fails?

Additional complexity for bonus points: (part of) the cluster might be hidden behind a proxy / load balancer requiring SNI, i.e. all (some) hostnames resolve to the same IP.

Side note: some of this may already be shimmable by individual callers, if they already know how they want things to connect: the Tunnel API allows callers to override how connections are established. This would, in theory, allow custom client-side code to take a requested endpoint and do whatever it needs, as long as it can create a Stream. I've seen this use proxy servers, custom DNS-like things, you name it.

I got a response from the team providing "redis as a service"... the summary is as follows:

The advertised hosts when receiving a MOVED response are correct [...]
the library should not use IPs from “CLUSTER NODES” responses for its own routing. We explicitly configure internal IPs for the Redis routing so that the sharding and replication does not go out the OpenShift network. We then expose different routing for client libraries as these need to be routable from outside the network. [...]
The documentation [...] states “Note that normally clients willing to fetch the map between Cluster hash slots and node addresses should use CLUSTER SLOTS instead.” The output of that command provides the hostname as first routing priority, and returns correct hostnames [...]
We will not be able to make the IPs routable from outside the OpenShift cluster

The output of CLUSTER SHARDS (7.x successor of CLUSTER SLOTS) is:

slots
 0
 5460
nodes
  id
  c9ef1500c2acc37273e5b04275f7e5620b796fff
  tls-port
  443
  ip
  10.42.0.1
  endpoint
  redis-7c2b3133-0bf4-0.redis.example
  hostname
  redis-7c2b3133-0bf4-0.redis.example
  role
  master
  replication-offset
  0
  health
  online
slots
 10923
 16383
nodes
  id
  753343875ffe163a64284b36d7743861f2ef1506
  tls-port
  443
  ip
  10.42.0.3
  endpoint
  redis-7c2b3133-0bf4-2.redis.example
  hostname
  redis-7c2b3133-0bf4-2.redis.example
  role
  master
  replication-offset
  0
  health
  online
slots
 5461
 10922
nodes
  id
  8992b30e1499e2cf0bc86609d752c4f3f49c2199
  tls-port
  443
  ip
  10.42.0.2
  endpoint
  redis-7c2b3133-0bf4-1.redis.example
  hostname
  redis-7c2b3133-0bf4-1.redis.example
  role
  master
  replication-offset
  0
  health
  online

Could you see a possible future for SE.R supporting such a topology / configuration?

@philon-msft

Well, we're making progress, at least. Yes, there is potential for us to migrate to CLUSTER SLOTS, but it'll take a bit of testing and effort. Basically, CLUSTER NODES was the original and therefore most widely available version of topology discovery. Moving to a newer version gets tricky because of server Vs client level, but CLUSTER SLOTS is almost as old as CLUSTER NODES so is almost certainly fine (IIRC both were pre-GA of 3.0). Migrating to CLUSTER SHARDS - probably not an option yet, and it looks like CLUSTER SLOTS is documented as supporting host names from v7, so: we're probably fine there.

But: this is at least actionable and makes reasonable sense. I don't think I can have much influence on the provider, but I suspect they may have similar problems with several client libraries, at least the cluster-aware ones, but: this seems pragmatic and I'm not going to fight making such a change. We might need to think about the possible matrix of server versions, host/IP configs, etc - the law of unintended consequences means that any such change that fixes some people: may also break others.

/Cc @philon-msft in case using hosts instead of IP routing might be useful, if/when we do this.

See edit for my proposal

Yes, CLUSTER SLOTS is returning the host names (together with optional, non-routable IP adresses), so seems to be the better choice:

0 0 0
  1 5460
  2 0 redis-7c2b3133-0bf4-0.redis.example
    1 443
    2 c9ef1500c2acc37273e5b04275f7e5620b796fff
    3 ip: 10.42.0.1
1 0 5461
  1 10922
 2 0 redis-7c2b3133-0bf4-1.redis.example
    1 443
    2 8992b30e1499e2cf0bc86609d752c4f3f49c2199
    3 ip: 10.42.0.2
2 0 10923
  1 16383
  2 0 redis-7c2b3133-0bf4-2.redis.example
    1 443
    2 753343875ffe163a64284b36d7743861f2ef1506
    3 ip: 10.42.0.3

Side note: in my case, all hostnames resolve to the same routable IP and require SNI.

Update; had a chat with the Redis Ltd folks today; the current consensus seems to be that this scenario is not a valid configuration, and that the host and advertised IP should map to the same. Still asking for more input from the Redis Ltd folks.

Reading the CLUSTER SLOTS documentation, and comparing the possible outputs, I am of the opinion that the "preferred endpoint" (the 0th element entry above) is meaningful sufficient to our purposes. Given that, I think I disagree with the Redis Ltd folks from yesterday. I think I'm going to try to advance this on the above basis, but probably with a new option along the lines of:

ClusterRouting: {Default,IP,Host}

where Default (the default) would use the "preferred endpoint", ip prefers ip (using host only if no ip available), Host prefers host (using ip only if no host available). Since CLUSTER SLOTS usually returns ip as the preferred endpoint, this represents no functional change in any relevant scenarios.

The above is open to discussion - that's just my current position!

ClusterRouting: {Default,IP,Host} (or even 'only' respecting the preferred order) seems good. If, for any reason, hosts / IPs are not reachable, an option to ForceX or XOnly would prevent polluting the candidate list... I guess this would already be possible using the mentioned Tunnel API?

For the record: I'm not using Redis in the aforementioned configuration anymore.

Toy server: node identity groundwork for endpoint reconciliation

ClusterNode: expose the endpoint trailer instead of truncating it

Add CLUSTER SLOTS: object model, parser, IServer surface, export

Preparation work for smart-client-handoffs (#3174)

Resolve servers by any identity a node answers to

Dial the advertised identity, and keep configured provenance under Re…

Route from CLUSTER SLOTS, and give endpoints a lifecycle (#3177)

hi is this resolved. we are also looking for this feature can contribute anywhere if required.

we are also exposing multiple shards through sni and cluster announce hostname. With other cluster-aware clients such as go-redis, Java clients, and redis-py, we can bootstrap using a single shard hostname and the client discovers/routs to the other shard hostnames correctly.

With StackExchange.Redis we were seeing MOVED errors when bootstrapping with a single hostname, while explicitly supplying all shard hostnames appears to avoid the issue.

Actually, it might be. The initial slots/identity implementation was in 3.2.0, but iirc it gets stronger in 3.3.0

Have you tried latest?

@mgravell

Yes — I tested with 3.3.0. The CLUSTER SLOTS hostname routing is working, but we still see an issue when bootstrapping with only a single DNS/SNI endpoint.

Our setup is:

Valkey Cluster with cluster-preferred-endpoint-type hostname

one announced hostname per shard

all shard hostnames resolve through a shared VIP / load balancer

routing to the correct backend shard depends on TLS SNI

CLUSTER SLOTS returns the shard DNS names as the preferred endpoints, while CLUSTER NODES still contains the internal/unroutable pod IPs.

With a single seed hostname, StackExchange.Redis appears to keep the seed hostname as SslHost, so connections to the discovered shard endpoints still use the original seed SNI. The load balancer then routes those connections to the wrong backend, and we eventually see errors such as:
MOVED ...
and in some cases:
InternalFailure / Command cannot be issued to a replica
If we explicitly configure all shard DNS names as endpoints, the issue goes away and our tests pass consistently.

So it looks like the remaining issue is specifically around per-discovered-endpoint SNI. For a DnsEndPoint, would it make sense for the TLS connection to use endpoint.Host as the SNI instead of reusing the global SslHost from the original seed?

@mgravell Could you please review this PR and let me know if the approach makes sense? #3250
I’ve tested the change against our internal Valkey cluster, where each shard is exposed through its own hostname and routing is based on SNI. With a single seed endpoint, the change correctly uses the discovered shard hostname for SNI, and cluster routing works as expected.

Closing as fixed by 3.2.0, with additional tweaks in #3250

Navigation Menu

Repository navigation

Cluster: support host names when available, via CLUSTER SLOTS #2826

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cluster: support host names when available, via CLUSTER SLOTS #2826

Description

Activity

CaringDev commented on Dec 19, 2024

mgravell commented on Dec 19, 2024

CaringDev commented on Jan 6, 2025

mgravell commented on Jan 6, 2025

mgravell commented on Jan 6, 2025

CaringDev commented on Jan 6, 2025

mgravell commented on Jan 16, 2025

mgravell commented on Jan 17, 2025

CaringDev commented on Jan 17, 2025

CaringDev commented on Apr 4, 2026

sandeepkunusoth commented on Sep 22, 2026

mgravell commented on Sep 22, 2026

sandeepkunusoth commented on Sep 22, 2026

mgravell commented on Sep 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions