[SteNicholas]

Purpose

Linked issue: #404 (follow-up of #461)

Since #461, the tsan-x86_64 job of Build and Test runs past its 120-minute timeout (run 41000500442): paimon-global-index-test grew from about 1400s to about 3500s.

Every LuceneGlobalIndexReader and LuceneGlobalIndexWriter constructed its own cppjieba::Jieba, reloading about 11.6MB of dictionaries each time. A table-level full-text search creates one reader per shard, so every query paid this cost once per shard, and the table-level full-text search tests added in #461 run many such queries.

This PR adds LuceneUtils::GetJieba():

• Each dictionary directory is loaded once per process, and a std::shared_ptr<const cppjieba::Jieba> is shared by all Lucene index readers and writers. The lock is held while loading, so concurrent first callers wait for a single load.
• Segmentation only calls const cppjieba methods, which keep no mutable state and are safe to call concurrently.
• The cache is keyed by the canonical absolute path. A relative PAIMON_JIEBA_DICT_DIR is resolved against the current working directory instead of reusing dictionaries loaded from another one, and a directory that cannot be resolved returns an IOError instead of aborting inside cppjieba.
• Trade-off: the dictionaries stay in memory until the process exits, so changes to the dictionary files take effect only after a restart.

Measured on the fork CI of the first revision of this change (cc54c144, before the path canonicalization):

Sanitizer job	Before #461	After #461	This PR
tsan-x86_64: paimon-global-index-test	1382s	3512s	156s
tsan-x86_64: whole job	passed	cancelled at 120 min	passed in 62 min
tsan-aarch64: paimon-global-index-test	1019s	3045s	103s

Tests

• LuceneUtilsTest.TestGetJiebaSharesOneSegmenter: concurrent first callers on a fresh dictionary directory get one segmenter and segment correctly with it; after all callers release their references, the cache still holds the segmenter and returns it again.
• LuceneUtilsTest.TestGetJiebaResolvesRelativeDictionaryDir: an unresolvable relative directory returns an error; the same relative directory resolved from two working directories loads two isolated segmenters (a user word changes the Cut result of one of them); the absolute path returns the cached segmenter.
• The existing LuceneGlobalIndexTest cases and the Lucene cases of GlobalIndexTest cover the reader and writer paths.

Validation: pre-commit passed on the changed files. The change was not built locally. On the fork, the first revision passed every completed job of Build and Test (including tsan-x86_64, tsan-aarch64, asan-ubsan-x86_64 and asan-ubsan-aarch64) and Gcc 8 Test; CI of the current revision is pending.

API and Format

No. LuceneUtils is internal; there are no changes under include/ and no storage format or protocol changes.

Documentation

No new feature. The description of PAIMON_JIEBA_DICT_DIR in docs/source/user_guide/global_index.rst now explains how a relative directory is resolved and that dictionary changes take effect only after the process restarts.

Generative AI tooling

Generated-by: Claude Code (Claude Opus 5.5)

🤖 Generated with Claude Code

[lxy-9602]

+1 Thanks for fixing ci!

[lucasfang]

+1

[lszskye]

+1