Skip to content

feat(cache): take maintenance notices from push frames as well as pub/sub - #196

Open
cosmin-staicu wants to merge 1 commit into
fix/stream-quarantine-two-cyclefrom
feat/maintenance-notifications-both
Open

cosmin-staicu wants to merge 1 commit into
fix/stream-quarantine-two-cyclefrom
feat/maintenance-notifications-both

Conversation

@cosmin-staicu

@cosmin-staicu cosmin-staicu commented Sep 19, 2026

Copy link
Copy Markdown
Member

Stacked on #195#194#193. Merge bottom-up; the diff here is the maintenance handling alone.

What was being dropped

RedisPlannedMaintenance.OnServerMaintenance matched one subtype and returned:

if (e is not AzureMaintenanceEvent azureEvent) { return; }

AzureMaintenanceEvent is the pub/sub notification Azure Cache for Redis publishes on AzureRedisEvents. Redis Enterprise and Redis Cloud send RESP3 push frames instead, which surface as PushMaintenanceEvent — a sibling under the same ServerMaintenanceEvent, and discarded at that type check.

docs/how-to/resilience.md already carried the consequence as a known limitation: "Azure Managed Redis (*.redis.azure.net) does not publish it, so on that service PlannedMaintenanceEnabled never fires and the planned-maintenance state never reports in-progress." That note is obsolete now, and this PR rewrites it.

Azure Managed Redis is recognised as a push provider, but its servers do not emit these frames yet (upstream support is pre-emptive), so nothing changes there today — the route is ready for when the rollout lands. The providers that do emit them are Redis Enterprise and Redis Cloud, and on those the discarded route was the only one that reached a consumer at all.

The two routes are treated differently

They are different situations, and conflating them would do harm:

Azure Cache for Redis Redis Enterprise / Redis Cloud
Arrives as AzureRedisEvents pub/sub RESP3 push frame
Server behaviour announces, hands nothing off relaxes timeouts, re-reads topology, re-subscribes moved sharded channels, moves off the endpoint
What we do probe every second, ForceReconnect() on failure record it; recovery is the client's

Probing on the push route would force a reconnect against a handoff already under way — fighting the mechanism the notification exists to enable. So that route is recorded and not acted on.

A notification is recorded whichever connection delivered it, and recognised if it arrives twice. It can arrive more than once: Azure's is a pub/sub broadcast every connection receives, and a push frame is replayed to a connection that reconnects, which the client collapses only within the multiplexer that received it. Deciding by connection state instead — is the other route subscribed? — answers about now rather than about when the message was published, and pub/sub does not replay, so a copy that connection never received is the only copy there was.

Both routes therefore record, and a copy matching one seen in the last 30 seconds is dropped, on the notification's own identity: for Azure's, the fields the client parsed out of the payload; for a push frame, the notification type and sequence id the client documents for exactly this. Nothing stamped per copy forms part of either key. A frame whose sequence could not be read is excluded — the client reports those as sequence zero and declines to collapse them itself, and zero is a legitimate sequence too, so the two are told apart by the description the client attaches.

Two asymmetries remain. A push frame arriving on the planned-maintenance connection is ignored, since that connection carries no commands. And on the command connection only a MOVING is tied to a generation — it names that connection's replacement — where the generation is the one carrying commands or the one about to, since a rebuild subscribes the replacement before publishing it and the server does not retain a MOVING for replay. The broadcast kinds are forwarded from a retired generation too, which may be the only one that saw them.

That needs the command multiplexer, which RedisPlannedMaintenance does not own — hence the new IRedisConnector.ServerMaintenance. The connector re-wires it on every multiplexer it builds, so a single subscription survives a ForceReconnect; subscribing to Database.Multiplexer directly would quietly stop working after the first rebuild. It is a defaulted no-op event, so no existing implementer breaks — the same shape as GetPrimaries() in #193.

Reporting an announced disruption through IRedisPlannedMaintenance.InProgress is #197, stacked on this. It needs a window whose lifecycle has to agree with the Azure probe loop it runs alongside, and that is where the concurrency lives. This change stands on its own without it, and InProgress stays driven by the Azure route here.

A source neither type models is now recorded from the base event rather than dropped. Matching one subtype and returning is exactly what hid the push kind for a year, and upstream explicitly invites other vendors to integrate.

Opting in

3.3.0 ships this off unless asked; auto-enlistment by provider hostname is stated upstream as direction, not as current behaviour. RedisConnectionOptions.MaintenanceNotifications is the opt-in — Auto to ask and connect normally if unsupported, Required to refuse a connection that will not deliver them (useful for proving it live in staging). null leaves the client's own default, so provider enlistment applies for free once it ships.

It takes RedisMaintenanceNotifications, an enum of our own, not StackExchange's. SER010 rides on their type, so exposing it would raise that diagnostic in every consumer that sets the option — unacceptable for a public package whose consumers may be on AWS ElastiCache, Valkey or self-hosted Redis and never receive one of these notifications at all. The suppression stops at our mapping.

One behaviour change worth review

A handoff raises ConnectionFailed with ConnectionFailureType.MaintenanceHandoff. That was tracked as Redis.ConnectionFailed — an alert on precisely the event advance notice exists to make uneventful. It is tracked as Redis.MaintenanceHandoff now. The event is still raised to OnConnectionFailed subscribers, because the connection genuinely did drop; only the telemetry name separates them.

RedisHealthCheck also stops naming Azure in its message, now that any provider can announce a window.

A throwing subscriber no longer costs the rest their notification

Ported from the review of UiPath/Orchestrator#50039, which hit the same shape.

RedisConnector and ConnectionStateMonitor raised their connection events with a plain multicast invoke, which stops at the first handler that throws. ForceReconnect caught the exception, but around the whole invocation list rather than around each handler, so the outcome was the same.

Reconnection is how RedisPubSubSubjectWriter, RedisStreamNotifyChannel and RedisStreamSubjectWriter re-subscribe, and the pub/sub writer's handler calls Timer.Change without the ObjectDisposedException guard its siblings carry, so a dispose racing the notification throws. One such throw left every subscriber after it in the list detached until the next reconnect.

RaiseIsolated walks the invocation list and tracks each handler's exception on its own. Two things this PR introduces made it worth doing here rather than separately:

  • ServerMaintenance had no guard at all. It is raised on StackExchange.Redis's own dispatch thread, and its production subscriber -- RedisPlannedMaintenance.OnServerMaintenance -- does dedup lookups, clock reads and telemetry, so a throw from it escaped into the client.
  • ConnectionStateMonitor's re-multicast sits inside the connector's, so a throw from one of its IConnectionState subscribers aborted the connector's list too.

The existing ForceReconnect_SwallowsOnReconnectedHandlerException registers one handler and asserts only that ForceReconnect completes, so it stayed green under the abort. Seven new tests -- four on the connector's events, three on the monitor's -- each register a throwing handler ahead of a counting one and assert the counting one ran and the exception was tracked. All seven were red-checked against the weakest implementation that still swallows (one try/catch around the whole loop), and only those seven failed.

Tests

RedisPlannedMaintenanceRoutingTests raises real events through the real subscription path (Raise.Event<EventHandler<ServerMaintenanceEvent>>), building the event types by reflection since their constructors are internal — the approach the repo already uses for StreamInfo/StreamConsumerInfo. It covers: an announced disruption recorded without reconnecting, a push frame arriving on the maintenance connection ignored, the notification recorded with its source, and an unmodelled source recorded rather than dropped. Verified red against the old discard-on-type-check.

Deduplication has its own group, each case red-checked against the mutation it exists to catch: the same broadcast on both routes, the same broadcast from two connection generations, a replayed push frame, a starter and its completion staying distinct, frames whose sequence could not be read each recorded, a genuine sequence zero collapsed, a copy still recognised behind 64 intervening notifications, a repeat past the retention recorded again, and the retention surviving a backward clock correction. Disposing_cancels_the_token_a_connection_attempt_was_given pins that shutdown ends the wait rather than leaving it suspended. Only the wait: ConnectionMultiplexerFactory checks the token once and then awaits ConnectionMultiplexer.ConnectAsync, which takes none, so a real attempt already in flight runs to completion.

RedisConnectorLifecycleTests.ServerMaintenance_IsForwarded_FromTheCurrentMultiplexer_AcrossAReconnect covers the wiring itself — the initial multiplexer, the replacement after a ForceReconnect, and the retired one, with teardown gated so the assertions do not race it. Without it every routing test would pass while production received nothing, since they all substitute the interface. AMoving_IsForwarded_FromTheReplacementBeforeItBecomesCurrent raises from inside the subscription itself, which is exactly the interval between the replacement being subscribed and being published.

ConnectionStateMonitorHandoffTests covers the monitor: a handoff forwards to subscribers exactly once while emitting Redis.MaintenanceHandoff and not Redis.ConnectionFailed, and a SocketFailure still reports as a failure. Verified red against the unconditional TrackEvent.

RedisConnectorTests pins the option mapping — Auto/Required/Disabled onto the client's modes, and null leaving the client's own default untouched. Required maps to Enabled rather than to a same-named member, so a reversed arm would leave the opt-in silently doing nothing.

Verification

  • Solution builds with 0 warnings.
  • Builds clean under the CI flags (-c Release -warnaserror).
  • Full suite against a live Redis: net10.0 1807/1807, net8.0 1786/1786.

Contributor declaration

  • I signed off my commits per the DCO (git commit -s).
  • I am contributing on behalf of my employer, or in the course of employment / using employer resources.

@github-actions github-actions Bot added the needs-cla-review A maintainer should assess whether a signed CLA is required (see CONTRIBUTING.md) label Sep 19, 2026
@github-actions

Copy link
Copy Markdown

🔎 Maintainer heads-up: automated triage flagged this PR as potentially material, so it may need a signed CLA in addition to the DCO sign-off.

Strong signals

  • adds public API surface (PublicAPI.Unshipped.txt in src/UiPath.Caching)

This is advisory only — the bot does not decide. Please judge against the CLA criteria (material, product-critical, patent-sensitive, corporate contributor, broad commercial use). Note that thresholds can be gamed by splitting PRs, so use your judgement.

  • If a CLA is needed → add the cla-required label (a contributor comment with signing steps is posted automatically).
  • If it is not needed → replace needs-cla-review with cla-not-required so later pushes don't re-flag it.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Maintenance state correlation, expiry, event source, and telemetry routing still have correctness gaps.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 5 Medium severity · 2 Low severity

Open (7)
What changed in this PR

Adds provider-neutral Redis maintenance handling for RESP3 push notifications while preserving Azure pub/sub behavior.

Changes:

  • Adds push-notification routing, maintenance windows, and opt-in configuration.
  • Separates maintenance handoff telemetry from connection failures.
  • Updates tests, public API records, documentation, and changelog.
File Description
tests/​UiPath.Caching.Tests/​Redis/​RedisPlannedMaintenanceRoutingTests.cs Tests push-event routing and window behavior.
src/​UiPath.Caching/​Redis/​RedisPlannedMaintenance.cs Handles multiple maintenance notification sources.
src/​UiPath.Caching/​Redis/​RedisMaintenanceNotifications.cs Defines the public opt-in modes.
src/​UiPath.Caching/​Redis/​RedisHealthCheck.cs Makes maintenance health text provider-neutral.
src/​UiPath.Caching/​Redis/​RedisConnector.cs Separates maintenance handoff telemetry.
src/​UiPath.Caching/​Redis/​RedisConnectionOptions.cs Adds maintenance notification configuration.
src/​UiPath.Caching/​Redis/​RedisConfigurationOptionsProvider.cs Maps public options to StackExchange.Redis.
src/​UiPath.Caching/​PublicAPI.Unshipped.txt Records the new public API.
docs/​how-to/​resilience.md Documents both maintenance routes.
CHANGELOG.md Describes the new behavior.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/UiPath.Caching/Redis/RedisConnector.cs Outdated
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated
Comment thread docs/how-to/resilience.md Outdated
Comment thread src/UiPath.Caching/Redis/RedisConnectionOptions.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread src/UiPath.Caching/Redis/RedisConfigurationOptionsProvider.cs Outdated
Comment thread CHANGELOG.md Outdated
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Invalid enum values are silently disabled, and documentation overstates current Azure Managed Redis support.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity · 1 Low severity

Open (2)
Resolved since last review (3)

Comment thread src/UiPath.Caching/Redis/RedisConfigurationOptionsProvider.cs Outdated
Comment thread docs/how-to/resilience.md Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The core behavior is consistent and well tested; remaining feedback is non-blocking documentation and telemetry polish.

Review effort: Balanced
Findings: 1 Low severity

Open (1)
Resolved since last review (2)

Comment thread CHANGELOG.md Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

Core behavior is well covered; remaining feedback concerns non-blocking documentation precision.

Review effort: Balanced
Findings: 2 Low severity

Open (2)
Resolved since last review (1)

Comment thread CHANGELOG.md Outdated
Comment thread docs/how-to/resilience.md Outdated
@cosmin-staicu
cosmin-staicu force-pushed the feat/maintenance-notifications-both branch 2 times, most recently from 6884564 to a87f356 Compare September 19, 2026 12:35
@cosmin-staicu
cosmin-staicu requested a balanced review from Copilot September 19, 2026 12:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Push-event filtering and deduplication can still discard valid maintenance notifications.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity · 2 Medium severity

Open (3)
Resolved since last review (3)

Comment thread src/UiPath.Caching/Redis/RedisConnector.cs Outdated
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Sequence-zero deduplication is incorrect, and a test helper leaves background initialization tasks suspended.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
Resolved since last review (3)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated
Comment thread tests/UiPath.Caching.Tests/Redis/RedisPlannedMaintenanceRoutingTests.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The reconnect and shutdown paths retain unresolved concurrency risks around maintenance event handling.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
Resolved since last review (2)

Comment thread src/UiPath.Caching/Redis/RedisConnector.cs Outdated
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Reconnect-time routing, cross-connection deduplication, and provider-specific behavior require final human validation.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Resolved since last review (2)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Provider-specific routing, deduplication, and reconnect concurrency require final human validation.

Review effort: Balanced
Findings: 1 Low severity

Open (1)
Resolved since last review (1)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Connection-generation routing and shutdown synchronization require human review, with a cancellation-disposal race still unresolved.

Review effort: Balanced
Findings: 1 High severity

Open (1)
Resolved since last review (1)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cross-multiplexer lifecycle behavior and unresolved deduplication concerns require changes and final human validation.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
Resolved since last review (1)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated
Comment thread tests/UiPath.Caching.Tests/Redis/RedisPlannedMaintenanceRoutingTests.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cross-connection routing, reconnect generations, and shutdown synchronization require final human review.

Review effort: Balanced
Findings: None

Resolved since last review (2)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Connection lifecycle and dispatch-thread routing are concurrency-sensitive, and exception isolation remains incomplete on the auxiliary route.

Review effort: Balanced
Findings: 1 High severity

Open (1)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Concurrency-sensitive lifecycle and deduplication changes require human validation, with telemetry failure paths still unresolved.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
Resolved since last review (1)
Previously missed (1)

In code that hasn't changed since last review

Medium severity Contain telemetry fallback failures

src/​UiPath.Caching/​EventHandlerExtensions.cs:26

If TrackException throws, this loop still stops before later subscribers run and the exception can escape onto StackExchange.Redis's dispatch thread. Because ICachingTelemetryProvider has no no-throw contract, contain failures from the telemetry fallback as well so RaiseIsolated preserves its isolation guarantee.

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Concurrent duplicate delivery can still lose a notification when the first recording attempt fails.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Resolved since last review (2)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Concurrent shutdown, reconnect routing, deduplication, and experimental client integration require changes and final human validation.

Review effort: Balanced
Findings: None

Resolved since last review (1)
Previously missed (2)

In code that hasn't changed since last review

Medium severity Mark service stopped before unsubscribing handlers

src/​UiPath.Caching/​Redis/​RedisPlannedMaintenance.cs:124

StopAsync unsubscribes from the connector before Cancel() marks the service stopped. A maintenance callback that was already dispatched can enter StartConnectionProbing during this window, observe _stopped == 0, and schedule probing after shutdown has begun. Mark and cancel the service first, then detach both event handlers so in-flight callbacks hit the existing stopped-state guard.

Low severity Align Azure Managed Redis push notification support documentation

docs/​reference/​settings.md:58

The Azure Managed Redis support boundary is inconsistent: this setting says its servers do not emit push notifications yet, and the changelog says support depends on a future rollout, while the PR description says Azure Managed Redis already sends PushMaintenanceEvent and explicit Auto/Required opt-in enables the route. Confirm the actual 3.3.0 behavior and align the user-facing documentation and release note; otherwise users cannot tell whether this option currently works for Azure Managed Redis.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cross-generation routing and concurrent cleanup require human review, and cleanup can still abort if telemetry reporting throws.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Concurrency-sensitive routing and lifecycle changes need human validation, and start telemetry can still produce an unmatched end event.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Resolved since last review (1)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Connection cancellation can deadlock, real in-flight connects ignore disposal, and multiplexer-generation routing retains an unresolved race.

Review effort: Balanced
Findings: 1 High severity · 2 Medium severity

Open (3)
Resolved since last review (1)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs
Comment thread src/UiPath.Caching/Redis/RedisConnector.cs Outdated
Comment thread tests/UiPath.Caching.Tests/Redis/RedisPlannedMaintenanceRoutingTests.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Background probe telemetry failures can still abort recovery or leave unobserved task faults.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
Resolved since last review (3)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cross-generation routing and shutdown synchronization need human validation, and malformed Azure notifications may currently be deduplicated incorrectly.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Resolved since last review (2)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Probe lifecycle races and incomplete future-notification deduplication can produce incorrect maintenance behavior.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 4 Medium severity

Open (4)
Resolved since last review (1)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs
Comment thread tests/UiPath.Caching.Tests/Redis/RedisPlannedMaintenanceRoutingTests.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Maintenance routing and deduplication still have correctness defects in supported edge cases.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
Resolved since last review (4)

Comment thread src/UiPath.Caching/Redis/RedisConnector.cs Outdated
Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Concurrent multiplexer lifecycle, deduplication, cancellation, and experimental notification behavior warrant final human validation.

Review effort: Balanced
Findings: None

Resolved since last review (2)

@cosmin-staicu

Copy link
Copy Markdown
Member Author

Note on the ConditionalWeakTable in RedisConnector, since the choice is not self-evident.

The maintenance handler used to be the method group OnInternalServerMaintenance. Method-group conversion yields an equal delegate, so -= detached it without anyone having to keep a reference. It is now a closure over the multiplexer it is attached to — that is what removed the dependency on the event's sender — and each subscription is therefore a distinct instance. Rebuilding one at unsubscribe time would compare unequal and silently leave the handler attached, so the exact instance has to be kept.

Why an ephemeron table rather than a Dictionary: the stored delegate captures the multiplexer it is keyed by, so the value references the key. In a plain dictionary held by a process-lifetime connector that is an uncollectable cycle — any multiplexer dropped without DisposeMultiplexer running, such as a candidate the swap rejected, stays pinned with its sockets. ConditionalWeakTable exists for exactly the value-references-key case and lets the pair be collected. It is also thread-safe, which the two call sites need: subscription runs on the reconnect task, removal can come from dispose or from closing a retired connection.

AddOrUpdate rather than Add because Add throws on a duplicate key, and re-configuring the same multiplexer should be a no-op rather than a crash.

@cosmin-staicu

cosmin-staicu commented Sep 22, 2026

Copy link
Copy Markdown
Member Author

Known limitation on the MOVING generation check, worth recording since the reachability is easy to get wrong.

public interface IConnectionGroup : IConnectionMultiplexer, so a MultiGroupMultiplexer is an IConnectionMultiplexer. This repository never constructs one, but RedisConnectionOptions.ConnectionFactory (Func<ConfigurationOptions, IConnectionMultiplexer>) and ConnectionMultiplexerFactoryType both accept one, so a consumer can hand us a group today. That is the path on which the sender-based drop fixed in 48740b3 was reachable — not a hypothetical topology.

What remains imprecise: the group multiplexer fans a subscriber delegate out to every member (member.Multiplexer.ServerMaintenanceEvent += value), and nothing in the group layer reacts to maintenance — selection is driven by health checks, circuit-breaker failures and manual failover, never by a notification. So a MOVING can arrive from a member that is not the active one, and we forward it rather than dropping it.

Cost of that: one extra Redis.Maintenance record here, and in #197 a health window that did not need opening. It fails safe — nothing is lost.

The obvious tightening is to compare PushMaintenanceEvent.EndPoint against GetEndPoints(), which on a group delegates to the active member only, so it would be exact for any sensible topology (two members on one endpoint is not rejected by the library but has no failover value). Deliberately not doing it: endpoint equality is a trap — IPEndPoint versus DnsEndPoint, normalisation differences — and a mismatch would drop the notification rather than over-forward it, reintroducing the silent loss this PR exists to remove, on plain single connections where the current code is already correct.

The better fix belongs upstream, and the gap is narrower than it looks. IConnectionGroup is public and already
declares ConnectionGroupMember? ActiveMember, and ConnectionGroupMember is a public type — but
ConnectionGroupMember.Multiplexer is internal, so a subscriber can obtain the active member and still not
compare it against the sender the event carried. Making that one property public would make the check exact
in a line:

if (muxer is IConnectionGroup group && group.ActiveMember?.Multiplexer != sender) { return; }

No endpoint comparison, no reflection.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Push-event telemetry currently emits an ambiguous, precision-losing received timestamp.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)

Comment thread src/UiPath.Caching/Redis/RedisPlannedMaintenance.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Attribute placement is incorrect, and the stated Azure Managed Redis rollout status is inconsistent.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity · 2 Low severity

Open (3)
Resolved since last review (1)

Comment thread src/UiPath.Caching/Redis/RedisConnector.cs Outdated
Comment thread docs/recipes/redis-health-check.md
Comment thread tests/UiPath.Caching.Tests/Redis/RedisPlannedMaintenanceRoutingTests.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

A throwing telemetry sink can still abort reconnect notification and retired-connection cleanup.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Resolved since last review (3)

Comment thread src/UiPath.Caching/Redis/RedisConnector.cs

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Telemetry exceptions can still bypass event isolation, skipping monitor subscribers or escaping the Redis dispatch thread.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
Resolved since last review (1)

Comment thread src/UiPath.Caching/Redis/ConnectionStateMonitor.cs
Comment thread src/UiPath.Caching/Redis/RedisConnector.cs Outdated
…/sub

RedisPlannedMaintenance matched AzureMaintenanceEvent and returned on anything else.
That subtype is the pub/sub notification Azure Cache for Redis publishes on
AzureRedisEvents. Redis Enterprise and Redis Cloud send RESP3 push frames instead,
which surface as PushMaintenanceEvent, and those fell through the type check and were
lost. resilience.md already carried the consequence as a known limitation. Azure
Managed Redis counts as that kind of provider too, though its servers do not emit the
frames yet.

Both are handled now, and a source neither subtype models is recorded from the base
event rather than dropped -- matching one subtype and returning is precisely what hid
the push kind, and upstream invites other vendors to add their own.

Each route is handled on the connection where it carries meaning. Azure's is a pub/sub
broadcast every connection receives, so it is taken on the planned-maintenance
connection and the command connection's copy ignored, which keeps a single notification
from counting twice. Push frames are per connection -- a MOVING names the replacement
for the connection it arrived on -- so they are taken from the one carrying commands,
reached through the new IRedisConnector.ServerMaintenance. The connector re-wires that
on every multiplexer it builds, so one subscription survives a ForceReconnect where
subscribing to the multiplexer directly would not.

They are recorded and not acted on. The client relaxes timeouts, re-reads topology and
moves off a departing endpoint by itself, so probing -- which force-reconnects on a
failed write -- would fight the very handoff it exists to survive. Reporting an
announced disruption through IRedisPlannedMaintenance.InProgress needs a window whose
lifecycle has to agree with the Azure probe loop it runs alongside, which is a change of
its own; InProgress stays driven by the Azure route here.

RedisConnectionOptions.MaintenanceNotifications is the opt-in, since 3.3.0 ships the
feature off unless asked and recognising a hostname does not amount to asking. It takes
RedisMaintenanceNotifications, an enum of our own: SER010 rides on StackExchange's type,
so exposing theirs would raise that diagnostic in every consumer that sets the option --
in a public library whose consumers may be on AWS, Valkey or self-hosted Redis and never
see one of these notifications at all. An unsupported value is rejected rather than
folded into Disabled, because silently not asking is indistinguishable from asking and
being refused.

A handoff raises ConnectionFailed with FailureType MaintenanceHandoff, which the
connector tracked as Redis.ConnectionFailed and ConnectionStateMonitor tracked again --
an alert on the event that advance notice exists to make uneventful. Both track it as
Redis.MaintenanceHandoff now, and it is still raised to subscribers, because the
connection did drop.

A connection event raised by this change, and the ones already here, now reach
every subscriber: RedisConnector and ConnectionStateMonitor invoked their handlers
as one multicast, which stops at the first that throws. ForceReconnect caught it,
but around the whole invocation list rather than each handler, so the outcome was
the same. Reconnection is how the pub/sub writer, the stream notify channel and the
stream subject writer re-subscribe, and the pub/sub writer can throw
ObjectDisposedException when a dispose races the notification -- one such throw left
every subscriber after it detached until the next reconnect.

IsolationExtensions.RaiseIsolated walks the invocation list and tracks each
exception on its own; TrackIsolationFailure guards the report itself, since a sink
that cannot take it would put the exception back on the path the boundary exists to
keep clear. Every catch in RedisConnector and RedisPlannedMaintenance that exists so
the surrounding work carries on now reports through it -- including GetVersion,
whose Lazy would otherwise have cached the failure for the life of the process.

RedisPlannedMaintenance also claimed a notification identity before recording it, so
a handler that threw suppressed every copy for 30 seconds while recording nothing;
the claim is now released unless the recording succeeded. A probe run marks itself
started only once it has announced that, so a refused announcement no longer kills
the worker before the probe loop. An Azure payload the client could not parse leaves
every field at its default, so those are recorded individually rather than sharing
one identity. And a connection candidate the swap rejected no longer stays the
newest multiplexer.

Signed-off-by: Cosmin Staicu <cosmin.staicu@uipath.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Concurrent multiplexer-generation routing and experimental upstream maintenance behavior warrant final human validation despite comprehensive tests.

Review effort: Balanced
Findings: None

Resolved since last review (2)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-not-required Maintainer reviewed: no CLA required for this contribution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants