Skip to content

[type:fix] release the redis client that is replaced - #7157

Merged
Aias00 merged 1 commit into
apache:masterfrom
HY-love-sleep:fix/redis-connection-factory-destroy
Sep 22, 2026
Merged

Aias00 merged 1 commit into
apache:masterfrom
HY-love-sleep:fix/redis-connection-factory-destroy

Conversation

@HY-love-sleep

Copy link
Copy Markdown
Contributor

Motivation

RedisConnectionFactory opens a lettuce connection pool in its constructor (afterPropertiesSet())
but had no lifecycle end, so every site that rebuilds its redis client after a configuration change
could only drop the previous one: its pool and its threads stayed alive. This is the change #7156 asks
for, and the debt the review of #7153 pointed at.

Fixes #7156.

What is added

  • RedisConnectionFactory implements DisposableBean: destroy() delegates to
    LettuceConnectionFactory.destroy(), which shuts the pool down. A static
    destroyQuietly(ReactiveRedisConnectionFactory) is added for the callers that keep only the reactive
    template: it ignores a factory that has no lifecycle and logs a failure instead of throwing, so a
    client that cannot be released never breaks the configuration update that replaces it.
  • The four sites that replace a redis client now release the previous one, after the new one is
    installed (so a request never sees a destroyed client):
    • AiTokenLimiterPluginHandler
    • RateLimiterPluginDataHandler
    • RedisCache.close(), which closed a connection but not the factory that owns the pool
    • SensitiveWordPluginDataHandler, which also destroys the client in removePlugin

Notes

  • The replaced client is destroyed immediately, like CachePluginDataHandler already does with
    lastCache.close(): a configuration change may therefore fail a request that is still holding the
    old client. An unchanged configuration does not rebuild anything, so the common path is untouched.
  • No behaviour of the plugins changes otherwise: the same client is created, cached and used.

Testing

./mvnw -pl shenyu-infra/shenyu-infra-redis,shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-sensitive-word,\
shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-token-limiter,\
shenyu-plugin/shenyu-plugin-fault-tolerance/shenyu-plugin-ratelimiter,\
shenyu-plugin/shenyu-plugin-cache/shenyu-plugin-cache-redis -am test
  • RedisConnectionFactoryTest: a real factory reports isRunning() and stops doing so after
    destroy(); destroyQuietly destroys a lettuce factory, ignores a factory without a lifecycle and a
    null one, and swallows a failing destroy().
  • The handler tests of the three plugins that cache a client now assert that changing the
    configuration destroys the replaced client while keeping the new one running, that an unchanged
    configuration does not rebuild it, and (for the sensitive word plugin) that removePlugin releases
    it.

- RedisConnectionFactory implements DisposableBean: destroy() delegates to the lettuce connection
  factory, and destroyQuietly(ReactiveRedisConnectionFactory) lets the callers that keep only the
  reactive template release the client they replace
- the four sites that rebuild a redis client on a configuration change now destroy the previous one:
  AiTokenLimiterPluginHandler, RateLimiterPluginDataHandler, RedisCache.close() (which closed a
  connection but not the factory that owns the pool) and SensitiveWordPluginDataHandler, which also
  releases the client in removePlugin
- tests: the lettuce factory stops running after destroy, destroyQuietly ignores a factory without a
  lifecycle (and a null one) and swallows a failure, and every handler destroys the client it replaces
  while keeping the new one running and without rebuilding an unchanged configuration
* was built for must not be used afterwards.
*/
@Override
public void destroy() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding the lifecycle end here is the right move: this wrapper is the only type that owns the two things LettuceConnectionFactory created but nobody released - the connection pool and the per-client Netty resources - so destroy() belongs to whoever created them.

I specifically checked the failure mode that would have made this dangerous: could releasing one Shenyu redis client take down another plugin's client because they share Lettuce ClientResources? It cannot, and the reasoning is worth writing down because it is load-bearing for every call site below:

  1. getLettuceClientConfiguration(...) builds LettucePoolingClientConfiguration.builder().poolConfig(...).build() and never calls .clientResources(...), so the configuration carries no resources.
  2. Lettuce 6.3.2, AbstractRedisClient constructor (decompiled from lettuce-core-6.3.2.RELEASE.jar): if (clientResources == null) { sharedResources = false; clientResources = DefaultClientResources.create(); } - each client creates and therefore owns its own resources.
  3. AbstractRedisClient.closeClientResources(...) only does the full clientResources.shutdown(...) when sharedResources == false; otherwise it just releases the event loop groups it borrowed.

So destroy() releases exactly one client's own resources and nothing else. Two further properties make it safe to call from the handlers: LettuceConnectionFactory.stop() only does work under state.compareAndSet(STARTED, STOPPING), so a second destroy() is a no-op, and isRunning() is a clean observable to assert on.

One note rather than a request: the class is not a Spring bean anywhere, and RedisConnectionFactory is not a ReactiveRedisConnectionFactory either, so destroyQuietly can never actually be handed one - every call site passes the inner LettuceConnectionFactory, which has always implemented DisposableBean on its own. Right now the instance destroy() has exactly one caller, its own unit test. It is still the right home for the lifecycle, but if you want it to earn its keep, the follow-up would be to have the call sites register themselves here instead of extracting the lettuce factory.

} catch (Exception ignored) {
}
// the factory owns the connection pool and its threads, closing a connection does not release them
RedisConnectionFactory.destroyQuietly(connectionFactory);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the right hook to release on. Worth being aware of how the caller sequences it, because this one is looser than the other three sites in this PR:

CachePluginDataHandler:

this.closeCacheIfNeed();                                            // destroys the old cache here
final ICacheBuilder cacheBuilder = ExtensionLoader...getJoin(...);
Singleton.INST.single(ICache.class, cacheBuilder.builderCache(config));  // new one installed later

The old hand - closeCacheIfNeed() - only does this:

ICache lastCache = CacheUtils.getCache();       // Singleton.INST.get(ICache.class)
...
lastCache.close();

It never removes ICache.class from Singleton.INST. So between the destroy above and the install below, CacheUtils.getCache() still hands out the destroyed cache, and after removePlugin it stays there indefinitely.

Before this change that window was survivable: close() only returned two borrowed connections to the pool, so a request that had already grabbed the cache still worked. Now the factory underneath it is destroyed, so that same request fails. Same class of window the other handlers have, but here it can be closed cheaply:

  • either install first, then close the previous one (what you did for the other three handlers), or
  • have closeCacheIfNeed() evict ICache.class right after lastCache.close().

Also happy to leave it as a follow-up issue if you prefer to keep this PR scoped to the leak - just say so and I will drop it. Flagging it because the cache plugin is the one replacement path where the old instance stays reachable after being destroyed.

Detail, no action needed: with pooling enabled, connectionFactory.getReactiveConnection() borrows a pooled connection and close() returns it, so the two lines above behave as intended rather than opening throwaway sockets.

Note on verification: I could not execute this one. RedisCacheTest is compiled and declares three @Test methods, but Surefire reports Tests run: 0 for it in this module (targeted and untargeted runs alike) on my machine - same before your change, so it is not something this PR introduced. I checked the existing closeCache() test by reading: its last close() runs against mocked factories, which are not DisposableBean, so destroyQuietly no-ops there, and the real cache is not used after close().

REDIS_PROPERTIES.get().cachedHandle(PLUGIN_NAME, redisConfig);
// the client that is replaced must not keep its connection pool and its threads alive
if (Objects.nonNull(cachedTemplate)) {
RedisConnectionFactory.destroyQuietly(cachedTemplate.getConnectionFactory());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Installing the new template with cachedHandle(...) before destroying the old one is exactly the right order - new requests can never observe a destroyed client. Capturing cachedTemplate before mutating the caches is also necessary, since after cachedHandle the handle is already the new template. Same thing in removePlugin just below, where destroying before removeHandle keeps the DTO reachable for the release.

Two things you may want to fold in later, neither blocking:

  1. With pooling configured (LettucePoolingClientConfiguration), what was actually being leaked here per config change was the pool plus one owned set of Netty event loops - see the comment on RedisConnectionFactory#destroy for why releasing it does not disturb the other plugins' clients. Would be nice to have a note here saying the released client must not be reused, e.g. by using it from a finally that swaps the reference.

  2. removePlugin now releases the client, which this handler did not do before - good. For symmetry, AiTokenLimiterPluginHandler and RateLimiterPluginDataHandler still have no removePlugin: those clients stay alive (with their pool and threads) for the rest of the process lifetime after the plugin is removed, because nothing ever clears their CommonHandleCache / Singleton entries. Not a regression introduced here, but it is the same bug through a different door, and now there is a destroyQuietly helper that makes the follow-up a two-line change per handler.

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Completes the RedisConnectionFactory lifecycle: destroy() delegates to LettuceConnectionFactory.destroy(), and the four sites that replace a Redis client on a configuration change now release the one they replace instead of dropping it. This closes #7156 and clears the debt the review of #7153 pointed at.

The thing I most wanted to rule out: shared client resources

The dangerous version of this change is "destroying one Shenyu Redis client kills another plugin's client because they share Lettuce ClientResources". It does not happen, and it is worth recording why:

  1. getLettuceClientConfiguration(...) builds LettucePoolingClientConfiguration.builder().poolConfig(...).build() and never passes .clientResources(...), so the configuration carries no resources.
  2. Lettuce 6.3.2, AbstractRedisClient constructor (decompiled from lettuce-core-6.3.2.RELEASE.jar): if (clientResources == null) { sharedResources = false; clientResources = DefaultClientResources.create(); } - each client therefore owns its own.
  3. AbstractRedisClient.closeClientResources(...) only does the full clientResources.shutdown(...) when sharedResources == false; otherwise it merely releases the event loop groups it borrowed.

So destroy() releases exactly one client's own pool and event loops. Two further properties make it safe from the handlers: LettuceConnectionFactory.stop() only acts under state.compareAndSet(STARTED, STOPPING), so repeat destruction is a no-op, and isRunning() is a clean observable to assert on - which is what the new tests do.

Verification

CI never runs unit tests on PRs in this repo - ci.yml gates test_group on github.event_name == 'push', and it was skipped on this run - so I built the branch locally:

mvn -pl shenyu-infra/shenyu-infra-redis,
        shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-sensitive-word,
        shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-token-limiter,
        shenyu-plugin/shenyu-plugin-fault-tolerance/shenyu-plugin-ratelimiter,
        shenyu-plugin/shenyu-plugin-cache/shenyu-plugin-cache-redis -am test

BUILD SUCCESS, checkstyle clean, no failing or erroring test class across the five modules, including:

  • RedisConnectionFactoryTest 6/6 (the three new ones included)
  • RateLimiterPluginDataHandlerTest 8/8, AiTokenLimiterPluginHandlerTest 4/4 (new file), SensitiveWordPluginDataHandlerTest 13/13
  • RedisRateLimiterTest 8/8 - relevant because it stuffs mocks into Singleton.INST under ReactiveRedisTemplate.class; it still passes, since the rebuild decision keys off RedisConfigProperties, which no other test writes. Worth knowing that coupling exists.

One gap I could not close by execution: RedisCacheTest reports Tests run: 0 in this module on my machine (also when targeted explicitly, and also before this change), so the RedisCache path is verified by reading rather than running. I checked its existing closeCache() case: its final close() runs against mocked factories, which are not DisposableBean, so destroyQuietly no-ops there, and the real cache is not touched after close().

What I like

  • Install-new-then-destroy-old ordering in all three handlers, so a request never picks up a destroyed client.
  • destroyQuietly swallowing failures: a client that cannot be released must not break the config push that replaces it.
  • AiTokenLimiterPluginHandlerTest - a genuinely new test class for a handler that had none.

Follow-ups (inline, none blocking)

  1. RedisCache / CachePluginDataHandler: closeCacheIfNeed() destroys the old cache but never evicts ICache.class from Singleton.INST, and installs the new one only afterwards. Previously that window was survivable (only borrowed connections were returned); now the cache handed out in that interval has a destroyed factory. Either install-then-close, or evict right after close.
  2. AiTokenLimiterPluginHandler / RateLimiterPluginDataHandler have no removePlugin - after plugin removal those clients keep their pool and threads for the process lifetime. Same bug through another door, now a two-line fix each.
  3. RedisConnectionFactory.destroy() currently has one caller, its own test - the class is neither a Spring bean nor a ReactiveRedisConnectionFactory, so nothing can hand one to destroyQuietly. Still the right home for the lifecycle; just noting it is currently aspirational.

Approving - the leak is real, the fix is correct, the shared-resources hazard I went looking for is not there, and the tests exercise the actual isRunning() transition rather than mocking it.

@Aias00
Aias00 merged commit 1a8ff42 into apache:master Sep 22, 2026
22 checks passed
@HY-love-sleep
HY-love-sleep deleted the fix/redis-connection-factory-destroy branch September 22, 2026 05:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] RedisConnectionFactory has no lifecycle end — replacing a redis client leaks the previous Lettuce pool

2 participants