Skip to content

fix(cache): read the quarantine timestamp back before deleting a group - #195

Open
cosmin-staicu wants to merge 1 commit into
chore/stackexchange-redis-3.3.0from
fix/stream-quarantine-two-cycle
Open

cosmin-staicu wants to merge 1 commit into
chore/stackexchange-redis-3.3.0from
fix/stream-quarantine-two-cycle

Conversation

@cosmin-staicu

@cosmin-staicu cosmin-staicu commented Sep 19, 2026

Copy link
Copy Markdown
Member

Stacked on #194, which is stacked on #193. Merge in that order; the diff here is the quarantine fix alone.

What is wrong

Quarantine in RedisStreamHealthMaintainer is a wait. A consumer group is quarantined by writing the current instant into a hash field, and deleted once MaintainerQuarantineInterval has passed. CheckEmptyStreamGroupAsync, the no-consumers path, did exactly that.

CheckStreamGroupWithConsumersAsync, the stale-LastDeliveredId path, wrote the same instant into the same field — and then branched on whether the field existed:

var quarantineValue = await Database.HashGetAsync(context.QuarantineKey, groupInfo.Name, CommandFlags.PreferReplica);
if (quarantineValue.HasValue)
{
    // delete now; the recorded instant is never parsed, never compared
}

The timestamp was written and never read back. Presence stood in for elapsed time, and the two are not the same thing:

  1. Overlapping passes shrink the wait to zero. LockTakeAsync holds the lock for MaintainerCheckInterval, and the only LockReleaseAsync is in the catch. A pass that finishes normally leaves the lock to expire, and expiry is on the same clock as the timer — so a pass that outruns its interval is still running when the next one starts. The first writes the quarantine record, the second reads it and deletes the group, same cycle. Nothing waited.
  2. With no overlap it was still the wrong interval. "Seen again" was the entire test, so the group died on the following cycle — MaintainerCheckInterval, 30 min by default — rather than after the documented MaintainerQuarantineInterval of 1 h.

What changed

Both paths now take the recorded instant from GetQuarantinedAtAsync and judge it with QuarantineElapsed. There is one copy of the rule where there were two, which is what let them drift apart in the first place. The delete log line carries the instant the group was quarantined, because "was in quarantine" no longer distinguishes anything worth logging.

Behaviour change for anyone watching this path: a group with consumers and a stale last-delivered-id now lives roughly one extra cycle before deletion, because it serves the whole quarantine interval instead of a single pass. That is the contract the option documents, and the one the no-consumers path has honoured all along.

The test that was supposed to catch it

Group_with_consumers_and_stale_LastDeliveredId_is_deleted_when_already_quarantined seeded the quarantine field with DateTimeOffset.UtcNow and asserted the group was deleted in that same pass. It pinned the defect in place rather than catching it — its sibling on the no-consumers path backdates past the interval, as it should.

It now backdates too, renamed ..._is_deleted_once_the_quarantine_has_elapsed. The discriminating case is new: Group_with_consumers_and_stale_LastDeliveredId_survives_a_quarantine_that_has_not_elapsed, written first and confirmed red against the old code. The backdated case passes either way, which is how presence-only deletion went unnoticed through review.

Docs

docs/how-to/broadcast.md described only the no-consumers condition for quarantine. It now names both conditions and says the interval runs from the recorded timestamp rather than from the pass that happens to observe it.

Verification

  • Solution builds with 0 warnings.
  • Full suite against a live Redis: net10.0 1776/1776, net8.0 1755/1755.

Contributor declaration

  • I signed off my commits per the DCO (git commit -s).
  • I am contributing on behalf of my employer, or in the course of employment / using employer resources.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The implementation consistently enforces the documented interval and includes focused regression coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Fixes stale consumer groups being deleted before completing the configured quarantine interval.

Changes:

  • Centralizes quarantine timestamp parsing and elapsed-time checks.
  • Adds regression coverage for elapsed and active quarantines.
  • Updates maintainer documentation and changelog.
File Description
RedisStreamHealthMaintainer.cs Enforces the full quarantine interval.
RedisStreamTopicMonitorTests.cs Covers active and elapsed quarantines.
docs/​how-to/​broadcast.md Documents both quarantine conditions.
CHANGELOG.md Records the defect and fix.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@cosmin-staicu
cosmin-staicu force-pushed the fix/stream-quarantine-two-cycle branch from 0693e7c to 5c4c077 Compare September 20, 2026 04:42
@cosmin-staicu
cosmin-staicu force-pushed the chore/stackexchange-redis-3.3.0 branch from 60fa6ec to 7620908 Compare September 20, 2026 05:14
@cosmin-staicu
cosmin-staicu force-pushed the fix/stream-quarantine-two-cycle branch 2 times, most recently from 03d6cd5 to 57b5a56 Compare September 20, 2026 13:38
@cosmin-staicu cosmin-staicu changed the title fix(cache): make the stale-last-delivered-id quarantine serve its interval fix(cache): read the quarantine timestamp back before deleting a group Sep 20, 2026
@cosmin-staicu
cosmin-staicu force-pushed the fix/stream-quarantine-two-cycle branch 3 times, most recently from b78c564 to 6d28636 Compare September 20, 2026 13:53
@cosmin-staicu
cosmin-staicu force-pushed the chore/stackexchange-redis-3.3.0 branch 2 times, most recently from 3101da0 to b396451 Compare September 21, 2026 05:02
@cosmin-staicu
cosmin-staicu force-pushed the fix/stream-quarantine-two-cycle branch from 6d28636 to 3ca1eea Compare September 21, 2026 05:02
@cosmin-staicu
cosmin-staicu force-pushed the chore/stackexchange-redis-3.3.0 branch from b396451 to c53cdeb Compare September 21, 2026 05:23
@cosmin-staicu
cosmin-staicu force-pushed the fix/stream-quarantine-two-cycle branch from 3ca1eea to 526da2c Compare September 21, 2026 05:23
@cosmin-staicu
cosmin-staicu force-pushed the chore/stackexchange-redis-3.3.0 branch from c53cdeb to c6aad67 Compare September 21, 2026 19:38
Quarantine is a wait. The maintainer writes the current instant into a hash field and
deletes the consumer group once MaintainerQuarantineInterval has passed.
CheckEmptyStreamGroupAsync, the no-consumers path, did that. The stale-last-delivered-id
path wrote the same instant into the same field and then branched on HasValue, so the
value it had just written was never read back.

Presence is not elapsed time. The maintenance lock is taken for MaintainerCheckInterval
and released only when a pass throws, so a pass that outruns that interval is still
running when the timer starts the next one. One pass writes the record, the other finds
it and deletes the group in the same cycle, and the wait is worth nothing. Even without
an overlap the group went one check interval after being recorded -- 30 min by default --
instead of the hour the option documents.

Both paths now read the instant through GetQuarantinedAtAsync and test it through
QuarantineElapsed, leaving one copy of the rule where there were two to fall out of step.
The delete log carries the instant the group was quarantined, since "was in quarantine"
no longer separates one case from another. A group with consumers and a stale
last-delivered-id therefore survives about one cycle longer than before, which is the
interval it was always meant to serve.

The test on this path wrote UtcNow into the quarantine field and asserted deletion in
that same pass, so it encoded the defect rather than caught it. It now backdates past the
interval, and a new test holds the case that tells the two behaviours apart: a record
written a moment ago must not delete.

Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>
@cosmin-staicu
cosmin-staicu force-pushed the fix/stream-quarantine-two-cycle branch from 526da2c to d9a6206 Compare September 21, 2026 19:40

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants