[type:fix] release the redis client that is replaced - #7157
Conversation
- RedisConnectionFactory implements DisposableBean: destroy() delegates to the lettuce connection factory, and destroyQuietly(ReactiveRedisConnectionFactory) lets the callers that keep only the reactive template release the client they replace - the four sites that rebuild a redis client on a configuration change now destroy the previous one: AiTokenLimiterPluginHandler, RateLimiterPluginDataHandler, RedisCache.close() (which closed a connection but not the factory that owns the pool) and SensitiveWordPluginDataHandler, which also releases the client in removePlugin - tests: the lettuce factory stops running after destroy, destroyQuietly ignores a factory without a lifecycle (and a null one) and swallows a failure, and every handler destroys the client it replaces while keeping the new one running and without rebuilding an unchanged configuration
| * was built for must not be used afterwards. | ||
| */ | ||
| @Override | ||
| public void destroy() { |
There was a problem hiding this comment.
Adding the lifecycle end here is the right move: this wrapper is the only type that owns the two things LettuceConnectionFactory created but nobody released - the connection pool and the per-client Netty resources - so destroy() belongs to whoever created them.
I specifically checked the failure mode that would have made this dangerous: could releasing one Shenyu redis client take down another plugin's client because they share Lettuce ClientResources? It cannot, and the reasoning is worth writing down because it is load-bearing for every call site below:
getLettuceClientConfiguration(...)buildsLettucePoolingClientConfiguration.builder().poolConfig(...).build()and never calls.clientResources(...), so the configuration carries no resources.- Lettuce 6.3.2,
AbstractRedisClientconstructor (decompiled fromlettuce-core-6.3.2.RELEASE.jar):if (clientResources == null) { sharedResources = false; clientResources = DefaultClientResources.create(); }- each client creates and therefore owns its own resources. AbstractRedisClient.closeClientResources(...)only does the fullclientResources.shutdown(...)whensharedResources == false; otherwise it just releases the event loop groups it borrowed.
So destroy() releases exactly one client's own resources and nothing else. Two further properties make it safe to call from the handlers: LettuceConnectionFactory.stop() only does work under state.compareAndSet(STARTED, STOPPING), so a second destroy() is a no-op, and isRunning() is a clean observable to assert on.
One note rather than a request: the class is not a Spring bean anywhere, and RedisConnectionFactory is not a ReactiveRedisConnectionFactory either, so destroyQuietly can never actually be handed one - every call site passes the inner LettuceConnectionFactory, which has always implemented DisposableBean on its own. Right now the instance destroy() has exactly one caller, its own unit test. It is still the right home for the lifecycle, but if you want it to earn its keep, the follow-up would be to have the call sites register themselves here instead of extracting the lettuce factory.
| } catch (Exception ignored) { | ||
| } | ||
| // the factory owns the connection pool and its threads, closing a connection does not release them | ||
| RedisConnectionFactory.destroyQuietly(connectionFactory); |
There was a problem hiding this comment.
This is the right hook to release on. Worth being aware of how the caller sequences it, because this one is looser than the other three sites in this PR:
CachePluginDataHandler:
this.closeCacheIfNeed(); // destroys the old cache here
final ICacheBuilder cacheBuilder = ExtensionLoader...getJoin(...);
Singleton.INST.single(ICache.class, cacheBuilder.builderCache(config)); // new one installed laterThe old hand - closeCacheIfNeed() - only does this:
ICache lastCache = CacheUtils.getCache(); // Singleton.INST.get(ICache.class)
...
lastCache.close();It never removes ICache.class from Singleton.INST. So between the destroy above and the install below, CacheUtils.getCache() still hands out the destroyed cache, and after removePlugin it stays there indefinitely.
Before this change that window was survivable: close() only returned two borrowed connections to the pool, so a request that had already grabbed the cache still worked. Now the factory underneath it is destroyed, so that same request fails. Same class of window the other handlers have, but here it can be closed cheaply:
- either install first, then close the previous one (what you did for the other three handlers), or
- have
closeCacheIfNeed()evictICache.classright afterlastCache.close().
Also happy to leave it as a follow-up issue if you prefer to keep this PR scoped to the leak - just say so and I will drop it. Flagging it because the cache plugin is the one replacement path where the old instance stays reachable after being destroyed.
Detail, no action needed: with pooling enabled, connectionFactory.getReactiveConnection() borrows a pooled connection and close() returns it, so the two lines above behave as intended rather than opening throwaway sockets.
Note on verification: I could not execute this one. RedisCacheTest is compiled and declares three @Test methods, but Surefire reports Tests run: 0 for it in this module (targeted and untargeted runs alike) on my machine - same before your change, so it is not something this PR introduced. I checked the existing closeCache() test by reading: its last close() runs against mocked factories, which are not DisposableBean, so destroyQuietly no-ops there, and the real cache is not used after close().
| REDIS_PROPERTIES.get().cachedHandle(PLUGIN_NAME, redisConfig); | ||
| // the client that is replaced must not keep its connection pool and its threads alive | ||
| if (Objects.nonNull(cachedTemplate)) { | ||
| RedisConnectionFactory.destroyQuietly(cachedTemplate.getConnectionFactory()); |
There was a problem hiding this comment.
Installing the new template with cachedHandle(...) before destroying the old one is exactly the right order - new requests can never observe a destroyed client. Capturing cachedTemplate before mutating the caches is also necessary, since after cachedHandle the handle is already the new template. Same thing in removePlugin just below, where destroying before removeHandle keeps the DTO reachable for the release.
Two things you may want to fold in later, neither blocking:
-
With pooling configured (
LettucePoolingClientConfiguration), what was actually being leaked here per config change was the pool plus one owned set of Netty event loops - see the comment onRedisConnectionFactory#destroyfor why releasing it does not disturb the other plugins' clients. Would be nice to have a note here saying the released client must not be reused, e.g. by using it from afinallythat swaps the reference. -
removePluginnow releases the client, which this handler did not do before - good. For symmetry,AiTokenLimiterPluginHandlerandRateLimiterPluginDataHandlerstill have noremovePlugin: those clients stay alive (with their pool and threads) for the rest of the process lifetime after the plugin is removed, because nothing ever clears theirCommonHandleCache/Singletonentries. Not a regression introduced here, but it is the same bug through a different door, and now there is adestroyQuietlyhelper that makes the follow-up a two-line change per handler.
Aias00
left a comment
There was a problem hiding this comment.
Summary
Completes the RedisConnectionFactory lifecycle: destroy() delegates to LettuceConnectionFactory.destroy(), and the four sites that replace a Redis client on a configuration change now release the one they replace instead of dropping it. This closes #7156 and clears the debt the review of #7153 pointed at.
The thing I most wanted to rule out: shared client resources
The dangerous version of this change is "destroying one Shenyu Redis client kills another plugin's client because they share Lettuce ClientResources". It does not happen, and it is worth recording why:
getLettuceClientConfiguration(...)buildsLettucePoolingClientConfiguration.builder().poolConfig(...).build()and never passes.clientResources(...), so the configuration carries no resources.- Lettuce 6.3.2,
AbstractRedisClientconstructor (decompiled fromlettuce-core-6.3.2.RELEASE.jar):if (clientResources == null) { sharedResources = false; clientResources = DefaultClientResources.create(); }- each client therefore owns its own. AbstractRedisClient.closeClientResources(...)only does the fullclientResources.shutdown(...)whensharedResources == false; otherwise it merely releases the event loop groups it borrowed.
So destroy() releases exactly one client's own pool and event loops. Two further properties make it safe from the handlers: LettuceConnectionFactory.stop() only acts under state.compareAndSet(STARTED, STOPPING), so repeat destruction is a no-op, and isRunning() is a clean observable to assert on - which is what the new tests do.
Verification
CI never runs unit tests on PRs in this repo - ci.yml gates test_group on github.event_name == 'push', and it was skipped on this run - so I built the branch locally:
mvn -pl shenyu-infra/shenyu-infra-redis,
shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-sensitive-word,
shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-token-limiter,
shenyu-plugin/shenyu-plugin-fault-tolerance/shenyu-plugin-ratelimiter,
shenyu-plugin/shenyu-plugin-cache/shenyu-plugin-cache-redis -am test
BUILD SUCCESS, checkstyle clean, no failing or erroring test class across the five modules, including:
RedisConnectionFactoryTest6/6 (the three new ones included)RateLimiterPluginDataHandlerTest8/8,AiTokenLimiterPluginHandlerTest4/4 (new file),SensitiveWordPluginDataHandlerTest13/13RedisRateLimiterTest8/8 - relevant because it stuffs mocks intoSingleton.INSTunderReactiveRedisTemplate.class; it still passes, since the rebuild decision keys offRedisConfigProperties, which no other test writes. Worth knowing that coupling exists.
One gap I could not close by execution: RedisCacheTest reports Tests run: 0 in this module on my machine (also when targeted explicitly, and also before this change), so the RedisCache path is verified by reading rather than running. I checked its existing closeCache() case: its final close() runs against mocked factories, which are not DisposableBean, so destroyQuietly no-ops there, and the real cache is not touched after close().
What I like
- Install-new-then-destroy-old ordering in all three handlers, so a request never picks up a destroyed client.
destroyQuietlyswallowing failures: a client that cannot be released must not break the config push that replaces it.AiTokenLimiterPluginHandlerTest- a genuinely new test class for a handler that had none.
Follow-ups (inline, none blocking)
RedisCache/CachePluginDataHandler:closeCacheIfNeed()destroys the old cache but never evictsICache.classfromSingleton.INST, and installs the new one only afterwards. Previously that window was survivable (only borrowed connections were returned); now the cache handed out in that interval has a destroyed factory. Either install-then-close, or evict right after close.AiTokenLimiterPluginHandler/RateLimiterPluginDataHandlerhave noremovePlugin- after plugin removal those clients keep their pool and threads for the process lifetime. Same bug through another door, now a two-line fix each.RedisConnectionFactory.destroy()currently has one caller, its own test - the class is neither a Spring bean nor aReactiveRedisConnectionFactory, so nothing can hand one todestroyQuietly. Still the right home for the lifecycle; just noting it is currently aspirational.
Approving - the leak is real, the fix is correct, the shared-resources hazard I went looking for is not there, and the tests exercise the actual isRunning() transition rather than mocking it.
Motivation
RedisConnectionFactoryopens a lettuce connection pool in its constructor (afterPropertiesSet())but had no lifecycle end, so every site that rebuilds its redis client after a configuration change
could only drop the previous one: its pool and its threads stayed alive. This is the change #7156 asks
for, and the debt the review of #7153 pointed at.
Fixes #7156.
What is added
RedisConnectionFactoryimplementsDisposableBean:destroy()delegates toLettuceConnectionFactory.destroy(), which shuts the pool down. A staticdestroyQuietly(ReactiveRedisConnectionFactory)is added for the callers that keep only the reactivetemplate: it ignores a factory that has no lifecycle and logs a failure instead of throwing, so a
client that cannot be released never breaks the configuration update that replaces it.
installed (so a request never sees a destroyed client):
AiTokenLimiterPluginHandlerRateLimiterPluginDataHandlerRedisCache.close(), which closed a connection but not the factory that owns the poolSensitiveWordPluginDataHandler, which also destroys the client inremovePluginNotes
CachePluginDataHandleralready does withlastCache.close(): a configuration change may therefore fail a request that is still holding theold client. An unchanged configuration does not rebuild anything, so the common path is untouched.
Testing
RedisConnectionFactoryTest: a real factory reportsisRunning()and stops doing so afterdestroy();destroyQuietlydestroys a lettuce factory, ignores a factory without a lifecycle and anullone, and swallows a failingdestroy().configuration destroys the replaced client while keeping the new one running, that an unchanged
configuration does not rebuild it, and (for the sensitive word plugin) that
removePluginreleasesit.