Under low-memory conditions, StackExchange.Redis can return incorrect results to simple StringGetAsync calls

This is another report of #2804

Environment details:

We're seeing this on StackExchange.Redis 2.8.16, but I've reproduced on 2.8.37.
We're running dockerized .NET 8 on linux FROM mcr.microsoft.com/dotnet/aspnet:8.0
I'm running against an AWS Elasticache Redis cluster with these options: clustercfg.mycluster.amazonaws.com:6379,password=REDACTED,ssl=True,abortConnect=False

I'm hoping I can help identify this issue because I've attached a reproduction to this issue. It's not a perfect repro - you may have to run it several times (15+ for at least 30 seconds each) before seeing the bug. It may also OOM and you may have to restart it.

The TLDR of the repro is:

I'm using 1000 async Tasks to call StringGetAsync and StringSetAsync with a GUID key and the same GUID as the value
I'm using a separate thread to consume as much process memory as I can by stuffing large int[]s into a static list.
There's one additional thread that's attempting to read and write large values (100KB) to and from redis
The process memory is limited to 2GB. That's configured in launchSettings.json, so you may need to make sure you're using the correct launch profile

When the bug occurs, most or all calls to StringGetAsync return a value from a different call. In some cases the connection heals itself, but in other cases (including in production) we've seen the connection stay broken despite the memory pressure going away, and the calls continue to be mixed up. In our production scenario, this resulted in mostly deserialization errors (since the data returned did not match the format we were expecting), but in some cases resulted in data being returned to clients that did not own that data (for mixups that happened to have the correct format)

The program will report exceptions from the GUID-writing processes and the large value writer. It will also report when the GUID-writing processes get values they do not expect. When this happens, it happens in big bursts:

Any help with this issue would be appreciated! From my experimentation it seems that HighIntegrity mode helps, but...why? How can memory pressure lead to what seems to be a protocol desync?

RedisRepro.zip

That's very weird. I seem to have secured some dedicated library time, so I'll see what I can find. Interesting!

I'm struggling with repro on this, which might be in part because I have a stupid amount of RAM on this machine; I have repro'd once in x64 and once on x86, but the error is... weird:

Example 1:

Received f3964cd1-37a4-4b0c-83bc-6243f2c22c44, but expected f3964cd1-37a4-4b0c-83bc-6243f2c22c44

Example 2:

Received 56b98a63-ab74-495b-8426-b50abb4e2b98, but expected 56b98a63-ab74-495b-8426-b50abb4e2b98
Received acf15a79-460f-4ef0-8b39-83d1490c23a0, but expected acf15a79-460f-4ef0-8b39-83d1490c23a0
Received ddd02278-307f-45bf-8ca8-4ed1def3b282, but expected ddd02278-307f-45bf-8ca8-4ed1def3b282

(the weird thing being: the left and right sides are equal; this is not an off-by-one desync failure)

To me this suggests some weird timing issue, however: I'm very concerned by whatever is going on here, and I'm continuing to investigate - I just wanted to post an update.

adding Example 3:

Received cea912c6-c865-4313-ba6d-1d15ff6bec18, but expected cea912c6-c865-4313-ba6d-1d15ff6bec18
Received b70a6d35-a047-4664-83a8-fd30054504b9, but expected b70a6d35-a047-4664-83a8-fd30054504b9
Received cd17c451-aecc-4085-921b-e701fd0ba9b6, but expected cd17c451-aecc-4085-921b-e701fd0ba9b6
Received 5d3effaf-2b4d-4d53-8e2a-d9eaa19c3cba, but expected 5d3effaf-2b4d-4d53-8e2a-d9eaa19c3cba

Thank you for the quick update! I don't recall seeing that behavior on my machine (windows 64 bit, 32GB RAM) and I can't imagine what would cause that!

I almost didn't post this because the repro is very inconsistent on my machine as well, and I tried playing with many different combinations of memory consumption / number of readers & writers / sizes of cache members and was never really able to get it to be more consistent. Let me know if you need any more details!

I would speculate that this is the cause of the thing that made us add "high integrity mode", so I'm very motivated to keep digging here

I am facing the same issue in one of my apps.
We are using StackExchange.Redis version 2.6.70.
The app is running on kubernetes with multiple pods.
The docker image is mcr.microsoft.com/dotnet/aspnet:6.0-alpine.

This did happenend on february of this year, but at the time we weren't able to pin point the issue as comming from the lib returning the wrong values, so we implemented some guards to throw exceptions, register logs and metrics to try catch the issue before it causes colateral damege as it did the first time. One of the guard checks if the values returning from redis matched the key that requested them.

Well, that guard blow up yesterday.

the first thing we see is a few System.OutOfMemoryException then, two minutes after we start seeing lots of StackExchange.Redis.RedisConnectionException:. This two variations:
StackExchange.Redis.RedisConnectionException: Unexpected response to UNLINK: SimpleString: string
StackExchange.Redis.RedisConnectionException: Unexpected response to TYPE: Integer: 1

There is also some deserialization errors, that probably happened when key request returned a json string that did not matched the target type.

The worst for us is that the majority of the gets this app does are for keys of the same type of item, so there is a lot of wrong responsed that actualy fit the target type on the deserialization, leading to a silent error (the guards mentioned above avoid this errors now).

At the time of the occurrence we had 6 pods of this app running, only one of them presented this behavior.

I am updating the lib to version 2.8.37 and setting the highIntegrity flag to true as mentioned on the issue #2804.

Question: when this happen again and the flag highIntegrity is enable, the connection will just throw a RedisConnectionException or the lib has specific exception type for this case? I would like to catch when this specific error occur to emit some warning so that we can take some action.

I'd need to check the exception.

Re the rest: I think we can conclude that there's definitely a gremlin in here somewhere. I'm working on this as one of my main things currently.

If there's anything I can do to help (maybe run some kind of debug build and give you some log output when I'm able to repro?) let me know. HighIntegrity is enabled on our environments but at this point I'm emotionally invested : )

Well: we've had difficulty with hard repro locally, but I have a plan; if you are currently able to observe the problem (albeit intermittently), maybe the most useful thing is to try future changes (when available) to see if it goes away!

Out of curiosity, any update?

Not at current time, but: this is a priority and understood as a serious item, and I'm in the process of changing roles (and employer) which should make fixing things like this a part of my day job, so: it will be addressed

At the risk of just adding more noise to this issue - I think I can confirm that HighIntegrity=true fixes this problem. We had a similar event happen in our prod environment, but this time we got:

stackexchange.redis.redisconnectionexception: ResponseIntegrityFailure on REDACTED.cache.amazonaws.com:6379/Interactive, Idle/ResponseSequenceCheck, last: GET, origin: ResponseSequenceCheck, outstanding: 55, last-read: 0s ago, last-write: 0s ago, unanswered-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 7 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.8.16.12844

Our monitoring that detects whether the redis response matches our expectations did not trip.

@mgravell

Any update? Cc: @mgravell

Is the assumption that the recent perf rewrite will solve it?

Yes. Exactly. The gnarly bug is somewhere is the guts of arenas or pipelines. The rewrite sidesteps all of that completely. Progress is going well - right now it is rock-solid stable and performance on single-connections, and I'm piecing together the routing core. I hope to have an early public build maybe this week.

If there any updates or ideas let me know please.

We've reproduced the issue few times on 2.8.24.
Dockerized .NET 8 app on linux FROM mcr.microsoft.com/dotnet/aspnet:8.0.20-alpine3.22.
Happens sporadically on 1-2 random pods and can go on for days. Restart helps.

I'm going to try:

update to the latest 2.10.1;
automatic Redis connection recreation (adding new multiplexors in pool and blocking "bad") as a way not to loose perf with "high integrity mode" option;
experiments with "high integrity mode".
Thanks!

@ABorovtsov

@ABorovtsov Did any of the mentioned steps help? I'm currently encountering the same issue as you described.

@mgravell

@mgravell any update on the big refactoring?

@tbajkacz

@tbajkacz I'm very confident that high integrity mode has fixed this in our environment. Instead of incorrect responses from the library, we see ResponseIntegrityFailure which is much better. It seems to recover after a short time period as well.

Also I'm so sorry @mgravell but I misclicked and accidentally closed this. Can you reopen??

@tbajkacz

@tbajkacz we've updated the driver to 2.10.1 and increased memory limit by 50 percent on k8s pods. No new repro after these. Therefore in our case no need to enable "high integrity mode" and pay for it with perf

As a rolling update: the work for this is genuine and increasingly stable. It is not feature complete (I know there are bugs in RESP3 handshakes and transactions, for example), but it passes many of the basic tests, and shows the performance and stability improvements we were hoping for.

This is #3014, for the curious; it is approaching 30kLoC, so it is a pretty big set of changes, so please understand that we want to be careful with internal QA etc before unleashing on an unsuspecting public. To help with this, I have significantly increased the unit testability, including creating a large number of new tests that validate the behaviour of all of the "read" processors, to check the before vs after behaviours in pure isolation, and to validate how they behave against a wide range of inputs - we even found some unrelated bugs in the process. Additionally, I plan to make use of #3018 to aggressively test a wider range of connection-related scenarios.

As a side note, we're also testing a range of more-exotic RESP3 scenarios that the old implementation simply doesn't handle - "attributes" and "streams"; these are largely theoretical RESP3 features - I have never seen them in the wild, and I'm not even sure the server can emit them, but we know for sure that the new implementation can handle them, so: yay!

This set of changes will become 3.0; there will be almost zero API changes for 3.0, with the intent being to minimize any upgrade pain - it should be a drop-in replacement. The only API changes again: relate to things we found from the new unit tests - you can see this here:

the extensive new unit tests showed that we weren't exposing a few LCS fields, so the existing LCS reply has been extended to include this missing data
there is a new TestHarness added; this is not intended for most users, hence the [SER005] "experimental" marker; however, this is a new API designed to allow extenal module developers (think "NRedisStack") to use the improved unit-testing features to validate their parsing and formatting

The CI pass is now green across the board except the "simulate connection failure" tests are currently inactive (needs some work):

(Ubuntu) 4851 tests were completed in 45s with 4742 passed, 0 failed and 109 skipped.

(Windows) 4851 tests were completed in 48s with 4727 passed, 0 failed and 124 skipped.

That deals only with the functional side; there's also a lot of other profiling and testing to be done. Initial results show great stability, and great pure read especially for large results (think LRANGE) but there is some packet fragmentation we need to resolve in the send pipe that adversely impacts small commands; this was expected, and there is a plan - initial step was simply to get everything working.

Update: V3 preview is here: #3044

@mgravell

@mgravell, sorry for the noise on this old thread. I searched among the announcements and couldn't find (not even sure if you're able to provide this info at all right now), but do we have an estimated time to completion for 3.0.0 full release?

currently juggling 8.8, active:active, and v3; "working on it" is the best I can do; the remaining work is tiny compared to the rest, I just need to get to it

Tear down connection on write failure to prevent queue desync (#3092)

Additional reproduction details / similar issue on v2.8.31

I'm experiencing similar symptoms to this issue.

Environment:

StackExchange.Redis: 2.8.31
.NET: ASP.NET Core 9.0
Orchestration: Kubernetes (EKS)
Redis: AWS ValKey (compatible with Redis)

Pod Resources:

resources:
  requests:
    cpu: "100m"
    memory: "256Mi"
  limits:
    cpu: "500m"
    memory: "512Mi"

Observed Behavior:

Under moderate to high load in UAT environment (works fine in lower-load QA), the service occasionally throws OutOfMemoryException.
At the same time, IDatabase.StringGetAsync(key) intermittently returns incorrect data for existing keys:
- Sometimes returns the string "OK"
- Sometimes returns Redis server info (e.g. the full output of INFO replication)
- However, when I manually inspect the same key (via redis-cli or another client), the value is correct JSON.
We use JSON serialization/deserialization for our cached model objects.
No calls to Execute*, ScriptEvaluate, or any low-level commands that would naturally return "OK" or INFO output.
ConnectionMultiplexer is registered as Singleton.
IDatabase is resolved as Transient via connection.GetDatabase() and injected into Scoped services.

Suspected Trigger:
The issue appears when the .NET process is under memory pressure (close to the 512Mi limit). This matches the "low-memory conditions" mentioned in this issue.

Thanks for the report. StackExchange.Redis 3.0 ships a substantial rewrite of the low-level IO core — a new RESPite-based stack that replaces Pipelines.Sockets.Unofficial and the old backlog/write loop — and we're confident this issue is addressed by that work.

Closing as resolved in 3.0. If you can still reproduce on 3.0.x, please comment and we'll reopen and investigate with fresh diagnostics.

Navigation Menu

Repository navigation

Under low-memory conditions, StackExchange.Redis can return incorrect results to simple StringGetAsync calls #2883

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Under low-memory conditions, StackExchange.Redis can return incorrect results to simple StringGetAsync calls #2883

Description

Activity

mgravell commented on May 15, 2025

mgravell commented on May 19, 2025

burns47 commented on May 19, 2025

mgravell commented on May 19, 2025

jamir-araujo commented on Jun 6, 2025

mgravell commented on Jun 6, 2025

burns47 commented on Jun 6, 2025

mgravell commented on Jun 7, 2025

petrroll commented on Jun 28, 2025

mgravell commented on Jun 29, 2025

burns47 commented on Jul 29, 2025

petrroll commented on Sep 8, 2025

mgravell commented on Sep 8, 2025

ABorovtsov commented on Dec 9, 2025

tbajkacz commented on Jan 28, 2026

petrroll commented on Jan 29, 2026

burns47 commented on Jan 29, 2026

ABorovtsov commented on Feb 3, 2026

mgravell commented on Feb 26, 2026

mgravell commented on Mar 26, 2026

victormagalhaess commented on Apr 29, 2026

mgravell commented on Apr 29, 2026

chaoyebugao commented on Jul 15, 2026

mgravell commented on Jul 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions