Skip to content

[type:fix] release the clients that are replaced or removed (#7163) - #7164

Merged
Aias00 merged 4 commits into
apache:masterfrom
HY-love-sleep:fix/cache-replacement-order
Sep 23, 2026
Merged

Aias00 merged 4 commits into
apache:masterfrom
HY-love-sleep:fix/cache-replacement-order

Conversation

@HY-love-sleep

Copy link
Copy Markdown
Contributor

Motivation

Follow-up of #7157, tracked by #7163. Two of its three items are fixed here; the third one needs a
decision first, see Not in this PR below.

1. CachePluginDataHandler: the new cache is installed before the previous one is closed

handlerPlugin(...) closed the previous cache first (closeCacheIfNeed()) and installed the new one
afterwards, while the Singleton entry keeps pointing at the old instance until that install. Between the
two statements CacheUtils.getCache() therefore handed out the cache whose connection factory had just
been released by #7157, so a request landing in that window failed. The new cache is now built and
installed first, and the previous one is closed after it — the order the three handlers changed by #7157
already use. closeCacheIfNeed() keeps its meaning for the disabled and removed paths, where no request
can reach the cache any more.

Evicting ICache.class instead (the other option raised in the review) is not possible today: Singleton
has no removal API, and the readers of CacheUtils.getCache() do not null-check it.

2. AiTokenLimiterPluginHandler releases its client in removePlugin

The handler keeps its client in a CommonHandleCache, so removing or disabling the plugin left the client,
its connection pool and its Netty event loops alive for the rest of the process. removePlugin now
destroys it and clears both caches, like SensitiveWordPluginDataHandler does since #7157.

Not in this PR

RateLimiterPluginDataHandler has the same gap, but its client lives in Singleton.INST and Singleton
has no removal API. Destroying it without evicting the entry would make the next handlerPlugin treat the
destroyed client as valid — it rebuilds only when RedisConfigProperties differs — and hand it to
RedisRateLimiter / ConcurrentRateLimiterAlgorithm, which do not null-check
Singleton.INST.get(ReactiveRedisTemplate.class). The eviction point has to be decided first, so I asked in
#7163 whether Singleton should grow a remove(Class) or this handler should keep its client in a
CommonHandleCache like the others.

Testing

./mvnw -pl shenyu-plugin/shenyu-plugin-cache/shenyu-plugin-cache-handler,\
shenyu-plugin/shenyu-plugin-ai/shenyu-plugin-ai-token-limiter -am test

BUILD SUCCESS, checkstyle clean:

  • CachePluginDataHandlerReplacementTest 1/1 — a new class that captures, from a close() callback, which
    cache CacheUtils.getCache() hands out while the previous one is being closed, and asserts that it is
    already the new one. It is a separate class on purpose: CachePluginDataHandlerTest starts an embedded
    redis in @BeforeAll and reports Tests run: 0 on arm64 macOS (also without this change), so a test
    added there would not have run.
  • AiTokenLimiterPluginHandlerTest 5/5, including the new removePlugin case: the released client stops
    running and both caches are cleared.

HY-love-sleep and others added 2 commits September 22, 2026 14:09
- CachePluginDataHandler: the previous cache was closed before the new one was installed, and the
  Singleton entry keeps pointing at the old instance, so a request landing in between was handed the
  cache whose connection factory had just been released. Install first, then close.
- AiTokenLimiterPluginHandler: add removePlugin, which destroys the cached client and clears both
  caches, so removing or disabling the plugin no longer leaves its pool and its threads alive
- tests: CachePluginDataHandlerReplacementTest observes the replacement order without depending on the
  embedded redis of the existing test class, and AiTokenLimiterPluginHandlerTest covers removePlugin

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — this closes the two release paths that were still open after #7157.

Thanks for picking these up. Both halves address real gaps:

AiTokenLimiterPluginHandler#removePlugin — without it, removing the plugin left the Lettuce client (connection pool plus its Netty resources) alive until the process exited, and left both REDIS_CACHED_HANDLE and REDIS_PROPERTIES_CACHED_HANDLE entries behind. Destroying before removing the handles is the correct order, and Objects.nonNull guards the case where the plugin was never initialised. The test asserts the thing that actually matters (isRunning() is false and the template is gone), rather than just counting destroy calls.

CachePluginDataHandler — installing the new cache before closing the previous one is the fix I was hoping for. With #7157 landed, a closed cache really is dead (its client is released), so the old order left a window where CacheUtils.getCache() could hand out an already-closed instance. Extracting closeCache(ICache) and reusing it from closeCacheIfNeed() keeps the disable/remove paths unchanged — I checked, closeCacheIfNeed() is still used by the disabled branch and by removePlugin, so it is not dead code.

CachePluginDataHandlerReplacementTest is a neat way to pin the ordering without depending on the embedded Redis: capturing what CacheUtils.getCache() returns inside close() is exactly the assertion needed, and it would fail under the old order.

Two follow-ups, both non-blocking:

  1. RateLimiterPluginDataHandler still has no removePlugin. It is the same client-lifecycle leak through a different door, and now that destroyQuietly exists it is the same two lines. Happy to see it here or in a separate PR.
  2. ApplicationConfigCache.getInstance().invalidateAll() is now called before the new cache is installed, but Singleton.INST.single(ICache.class, ...) happens after — fine — though note that a request running concurrently between the two can still observe the old cache instance while its config cache has already been invalidated. That was true before as well; just flagging that this PR narrows the dangerous window but does not close it entirely.

CI note: the four shenyu-integrated-test-k8s-ingress-* jobs are red, but they fail with mvnd StaleAddressException: ... daemon may have crashed / No message received within 3000ms — build-daemon infrastructure, not this change. build, Analyze (java), CodeQL and check-license-header are all green.

@Aias00
Aias00 merged commit 41854f0 into apache:master Sep 23, 2026
24 of 27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants