fix(cache): read the quarantine timestamp back before deleting a group - #195
Open
cosmin-staicu wants to merge 1 commit into
Open
cosmin-staicu wants to merge 1 commit into
cosmin-staicu wants to merge 1 commit into
Conversation
cosmin-staicu
requested review from
alinahornet,
cosminvlad,
litheon,
lucianaparaschivei and
razvalex
as code owners
September 19, 2026 09:11
There was a problem hiding this comment.
Copilot review overview
🟢 Approval recommended
The implementation consistently enforces the documented interval and includes focused regression coverage.
Review effort: Balanced
Findings: None
What changed in this PR
Fixes stale consumer groups being deleted before completing the configured quarantine interval.
Changes:
- Centralizes quarantine timestamp parsing and elapsed-time checks.
- Adds regression coverage for elapsed and active quarantines.
- Updates maintainer documentation and changelog.
| File | Description |
|---|---|
RedisStreamHealthMaintainer.cs |
Enforces the full quarantine interval. |
RedisStreamTopicMonitorTests.cs |
Covers active and elapsed quarantines. |
docs/how-to/broadcast.md |
Documents both quarantine conditions. |
CHANGELOG.md |
Records the defect and fix. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This was referenced Sep 19, 2026
cosmin-staicu
force-pushed
the
fix/stream-quarantine-two-cycle
branch
from
September 20, 2026 04:42
0693e7c to
5c4c077
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.3.0
branch
from
September 20, 2026 05:14
60fa6ec to
7620908
Compare
cosmin-staicu
force-pushed
the
fix/stream-quarantine-two-cycle
branch
2 times, most recently
from
September 20, 2026 13:38
03d6cd5 to
57b5a56
Compare
cosmin-staicu
force-pushed
the
fix/stream-quarantine-two-cycle
branch
3 times, most recently
from
September 20, 2026 13:53
b78c564 to
6d28636
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.3.0
branch
2 times, most recently
from
September 21, 2026 05:02
3101da0 to
b396451
Compare
cosmin-staicu
force-pushed
the
fix/stream-quarantine-two-cycle
branch
from
September 21, 2026 05:02
6d28636 to
3ca1eea
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.3.0
branch
from
September 21, 2026 05:23
b396451 to
c53cdeb
Compare
cosmin-staicu
force-pushed
the
fix/stream-quarantine-two-cycle
branch
from
September 21, 2026 05:23
3ca1eea to
526da2c
Compare
cosmin-staicu
force-pushed
the
chore/stackexchange-redis-3.3.0
branch
from
September 21, 2026 19:38
c53cdeb to
c6aad67
Compare
Quarantine is a wait. The maintainer writes the current instant into a hash field and deletes the consumer group once MaintainerQuarantineInterval has passed. CheckEmptyStreamGroupAsync, the no-consumers path, did that. The stale-last-delivered-id path wrote the same instant into the same field and then branched on HasValue, so the value it had just written was never read back. Presence is not elapsed time. The maintenance lock is taken for MaintainerCheckInterval and released only when a pass throws, so a pass that outruns that interval is still running when the timer starts the next one. One pass writes the record, the other finds it and deletes the group in the same cycle, and the wait is worth nothing. Even without an overlap the group went one check interval after being recorded -- 30 min by default -- instead of the hour the option documents. Both paths now read the instant through GetQuarantinedAtAsync and test it through QuarantineElapsed, leaving one copy of the rule where there were two to fall out of step. The delete log carries the instant the group was quarantined, since "was in quarantine" no longer separates one case from another. A group with consumers and a stale last-delivered-id therefore survives about one cycle longer than before, which is the interval it was always meant to serve. The test on this path wrote UtcNow into the quarantine field and asserted deletion in that same pass, so it encoded the defect rather than caught it. It now backdates past the interval, and a new test holds the case that tells the two behaviours apart: a record written a moment ago must not delete. Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
cosmin-staicu
force-pushed
the
fix/stream-quarantine-two-cycle
branch
from
September 21, 2026 19:40
526da2c to
d9a6206
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #194, which is stacked on #193. Merge in that order; the diff here is the quarantine fix alone.
What is wrong
Quarantine in
RedisStreamHealthMaintaineris a wait. A consumer group is quarantined by writing the current instant into a hash field, and deleted onceMaintainerQuarantineIntervalhas passed.CheckEmptyStreamGroupAsync, the no-consumers path, did exactly that.CheckStreamGroupWithConsumersAsync, the stale-LastDeliveredIdpath, wrote the same instant into the same field — and then branched on whether the field existed:The timestamp was written and never read back. Presence stood in for elapsed time, and the two are not the same thing:
LockTakeAsyncholds the lock forMaintainerCheckInterval, and the onlyLockReleaseAsyncis in thecatch. A pass that finishes normally leaves the lock to expire, and expiry is on the same clock as the timer — so a pass that outruns its interval is still running when the next one starts. The first writes the quarantine record, the second reads it and deletes the group, same cycle. Nothing waited.MaintainerCheckInterval, 30 min by default — rather than after the documentedMaintainerQuarantineIntervalof 1 h.What changed
Both paths now take the recorded instant from
GetQuarantinedAtAsyncand judge it withQuarantineElapsed. There is one copy of the rule where there were two, which is what let them drift apart in the first place. The delete log line carries the instant the group was quarantined, because "was in quarantine" no longer distinguishes anything worth logging.Behaviour change for anyone watching this path: a group with consumers and a stale last-delivered-id now lives roughly one extra cycle before deletion, because it serves the whole quarantine interval instead of a single pass. That is the contract the option documents, and the one the no-consumers path has honoured all along.
The test that was supposed to catch it
Group_with_consumers_and_stale_LastDeliveredId_is_deleted_when_already_quarantinedseeded the quarantine field withDateTimeOffset.UtcNowand asserted the group was deleted in that same pass. It pinned the defect in place rather than catching it — its sibling on the no-consumers path backdates past the interval, as it should.It now backdates too, renamed
..._is_deleted_once_the_quarantine_has_elapsed. The discriminating case is new:Group_with_consumers_and_stale_LastDeliveredId_survives_a_quarantine_that_has_not_elapsed, written first and confirmed red against the old code. The backdated case passes either way, which is how presence-only deletion went unnoticed through review.Docs
docs/how-to/broadcast.mddescribed only the no-consumers condition for quarantine. It now names both conditions and says the interval runs from the recorded timestamp rather than from the pass that happens to observe it.Verification
Contributor declaration
git commit -s).