Skip to content

tenants: shut the delayed tenant provisioning down with its context - #7693

Merged
iliyan-velichkov merged 1 commit into
masterfrom
tenants-initializer-executor-lifecycle
Oct 5, 2026
Merged

iliyan-velichkov merged 1 commit into
masterfrom
tenants-initializer-executor-lifecycle

Conversation

@iliyan-velichkov

Copy link
Copy Markdown
Contributor

Cause

The PostgreSQL slow shard of master and the nightly has been failing with the same cascade (runs 37275439965, 37088834792, 36903920667). Every class after LocalNativeAppLifecycleIT (IntentEngineIT, ActAsSessionIT, ExternalFrontendTenantSelectionIT, ... 170 errors) fails after ~308 s with

liquibaseSystemDB: liquibase.exception.LockException: Could not acquire change log lock.
Currently locked by runnervm8df0l (10.1.0.211) since 10/5/26, 7:29 AM

TenantsInitializer scheduled the tenant provisioning 30 s after ApplicationReadyEvent on a ScheduledThreadPoolExecutor it never shut down. LocalNativeAppLifecycleIT boots one context per test method, each living about 13 s, so every one of its contexts left a provisioning timer behind. Those timers fired after their context had closed, five of them during the class, each on a pool-N-thread-1 thread. A late run goes through the closed context's beans, and Spring re-creates them on demand. The log shows a new SystemDB Hikari pool and the SystemDB Liquibase update running on that pool thread (07:28:21 [pool-13-thread-1] liquibase.changelog - Reading from public.databasechangelog). The last one fired at 07:29:25.206, 0.1 s after DirigibleCleaner had dropped and recreated the PostgreSQL system schema. The next context's Liquibase then found the changelog lock held and waited for it, and so did every class after it. On H2 the schema lives in a file under the fork's own target/, which is why only the PostgreSQL leg shows it.

The same leak exists outside tests: any context that closes within 30 s of becoming ready keeps a timer that later reaches into its closed beans.

Change

The executor is now a field of the bean and is shut down in destroy(). A provisioning that has not started when its context closes never runs. The delay is still 30 s. A package-private constructor takes the delay so the unit test does not have to wait 30 s.

Verification

  • mvn -pl components/core/core-tenants test: 33/33 green, including the new TenantsInitializerTest. It checks that the provisioning runs after the delay, and that a provisioning scheduled before destroy() never runs. Mutation check: with destroy() turned into a no-op, the second test fails.
  • PostgreSQL 16 locally (system DB and default DB as separate databases, as on the CI leg): -Dit.test=LocalNativeAppLifecycleIT,ActAsSessionIT -Dfailsafe.runOrder=reversealphabetical, i.e. ActAsSessionIT booting right after LocalNativeAppLifecycleIT, which is the order that fails on CI. Result: 10/10 green and BUILD SUCCESS. The log has no TenantsInitializer run after a context closed, no Liquibase on a non-main thread and no Waiting for changelog lock. For comparison, the CI log of the same class on master has five late provisioning runs.
  • formatter:validate on the module, with the cache wiped.
  • Not verified: the full PostgreSQL slow shard. The timing that poisons the lock depends on the runner, so the shard's next run on master is the real proof.

The other master failure, the api shards cancelled at their 90-minute cap, is unrelated and is addressed separately.

🤖 Generated with Claude Code

TenantsInitializer scheduled the provisioning 30 s after ApplicationReady on a
ScheduledThreadPoolExecutor it never shut down. A context closed within those
30 s still ran it later, through the closed context's beans, which Spring
re-creates on demand - a new SystemDB pool, an EntityManagerFactory and the
SystemDB Liquibase update - against a database that by then belonged to the
next context. On the PostgreSQL CI leg the one LocalNativeAppLifecycleIT left
behind fired 0.3 s after DirigibleCleaner had wiped the system schema and left
the Liquibase changelog lock held: every later IT class in the slow shard then
waited five minutes for the lock and failed to load its context.

The executor is now a field of the bean and shut down in destroy(), so a
provisioning that has not started when its context closes never runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iliyan-velichkov iliyan-velichkov self-assigned this Oct 5, 2026
@iliyan-velichkov
iliyan-velichkov merged commit cda1b6f into master Oct 5, 2026
10 checks passed
@iliyan-velichkov
iliyan-velichkov deleted the tenants-initializer-executor-lifecycle branch October 5, 2026 14:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant