feat(global-index): support table-level full-text search for data-evolution tables by SteNicholas · Pull Request #461 · apache/paimon-cpp

SteNicholas · 2026-10-24T14:01:25Z

Purpose

Linked issue: close #404

Paimon C++ only exposed the reader primitive for full-text search: a caller had to call
GlobalIndexScan::CreateReaders, run VisitFullTextSearch on the readers and pass the result to
ScanContextBuilder::SetGlobalIndexResult, choosing index types and shard ranges along the way,
with no serializable plan for running scan and read in different processes. This adds the
table-level API, aligned with Java FullTextSearchBuilder, DataEvolutionFullTextScan and
DataEvolutionFullTextRead.

FullTextSearchBuilder: SetQuery, SetLimit, SetPartitionFilter,
AddOption/SetOptions, WithMemoryPool/WithExecutor/WithFileSystem, NewFullTextScan(),
NewFullTextRead() and ExecuteLocal(). As in Java, the query column must exist in the latest
schema and a read requires a positive limit. A column with a primary-key full-text index
(pk-full-text.index.columns) is rejected with NotImplemented for now; Java searches it with
PrimaryKeyFullTextScan and PrimaryKeyFullTextRead.
Search modes: the new full-text-index.search-mode and global-index.search-mode options are
resolved as Java CoreOptions#fullTextIndexSearchMode does (the full-text option, or else the
global one, or else fast). Only the rows that an index shard covers are searched, as in fast,
so the builder rejects full and detail with NotImplemented until [Feature] Support full-text search modes for unindexed row ranges #406 instead of silently
leaving out rows written since the index was built.
DataEvolutionFullTextScan: first checks the time-travel options against scan.mode with
SchemaValidation::ValidateStartupMode, now implemented as Java
SchemaValidation#validateStartupMode (it was a NotImplemented stub), so for example
scan.snapshot-id is rejected with latest-full. It then resolves the snapshot as Java
TimeTravelUtil#tryTravelOrLatest does: from scan.snapshot-id, scan.tag-name, scan.timestamp
or scan.timestamp-millis, or else the latest one. The column, partition keys and options still
come from the latest schema, not from that of the snapshot (TODO). The scan keeps the index files
that pass the partition filter, index the column as their indexed field or as an extra field, and
whose indexer supports full-text search. As Java chooseIndexRanges does ([global-index] Fix full-text range selection and scalar-filter coverage for multi-field indexes paimon#8547), it
gives every row to one shard across overlapping and multi-field indexes, a dedicated index first,
then by index identity, start and end, and plans one split per chosen shard with its physical
range and its assigned search row ranges. A full-text or lucene-fts index whose backend is not
built fails the plan instead of silently dropping its rows.
DataEvolutionFullTextRead: reads each split with an indexer created from its own index
files, including their extra fields, through OffsetGlobalIndexReader, ranking only its search
row ranges and live rows, and asks each split for limit candidates only ([core] Request only top-k candidates from data-evolution full-text indexes paimon#9955).
Splits are searched in parallel on the given executor, or on one with global-index.thread-num
threads. The results are merged, keeping the first score of a duplicate row id, and cut to the
global top-k by score descending, then row id ascending. Live rows are pinned to the snapshot the
splits were planned from ([core] Support full-text indexes on DV tables paimon#9031), which is loaded from a tag once it has expired.
FullTextSearchSplit / IndexFullTextSearchSplit: carry the snapshot id, column, physical
range, search row ranges and index files, with a versioned encoding specific to Paimon C++. Java
serializes IndexFullTextSearchSplit with Java object serialization and pypaimon has no
serialized form, so there is no cross-language format to match, as [Feature] Support table-level full-text search for data-evolution tables #404 notes. Deserialization
rejects a null buffer and checks every count, range and binary row layout before decoding it, and
the read rejects splits from different snapshots, of another column, or whose index files no
longer index the field id of the column, such as after it was replaced.
GlobalIndexer::SupportsFullTextSearch(): false by default and true for full-text and
lucene-fts, as Java GlobalIndexerFactory#supportsFullTextSearch.
The scored top-k merge (ScoredGlobalIndexResultUtils::MergeTopK), the row id range helpers
(GlobalIndexUtils::RowIdsOf / RowIdsWithin) and executor creation
(GlobalIndexScanImpl::GetOrCreateExecutor) are shared with the existing union reader and scan.
The merge is also compiled into paimon_global_index, whose union reader calls it.

Not part of this change:

Row filters (withFilter, [Feature] Support row filters in full-text search #405), including the scalar index files that version 2 of Java
IndexFullTextSearchSplit carries to pre-filter rows (TODO at the split version).
The full and detail search modes ([Feature] Support full-text search modes for unindexed row ranges #406), which also search the rows that no index covers.
Searching with the schema of a time-travel snapshot (TODO in the scan).

Tests

UT:

src/paimon/core/table/source/data_evolution_full_text_scan_test.cpp:
TestIndexIdentityAndIndexedFields, TestDisjointShards (including a shard ending at
INT64_MAX), TestOverlappingShardsOfOneIndex, TestDedicatedIndexBeforeExtraField,
TestIdentityOrderBreaksTies and TestServesFullTextSearch.
src/paimon/core/table/source/data_evolution_full_text_read_test.cpp, with a fake full-text
indexer that records each search: TestMergesShardResultsIntoGlobalTopK,
TestRanksOnlySearchRowRanges, TestIncludeRowIds, TestReadsEachSplitWithItsOwnIndexFields,
TestRejectsInvalidSplits, TestRejectsSplitOfReplacedColumn, TestBuilderValidatesSettings
(including a rejected full search mode), TestBuilderRejectsPrimaryKeyFullTextIndex and
TestResolvesPlanSnapshot.
src/paimon/core/table/source/full_text_search_split_test.cpp: TestSerializeRoundTrip (the
complete version 1 bytes are pinned, with the index file entries cross-checked against Java
IndexFileMetaSerializer), TestDeserializeRejectsInvalidBytes,
TestDeserializeRejectsCorruptFileLayout, TestDeserializeRejectsCorruptEntries and
TestCreateValidatesArguments.
src/paimon/core/schema/schema_validation_test.cpp: TestStartupMode, every scan.mode with
missing, conflicting and valid time-travel options.
src/paimon/core/core_options_test.cpp: TestParseFullTextIndexSearchMode, the fallback to
global-index.search-mode, case-insensitive values and invalid values.
src/paimon/common/global_index/global_index_utils_test.cpp: TestRowIdsOfAndWithin.
SupportsFullTextSearch() is checked in global_indexer_factory_test.cpp (bitmap),
full_text_global_index_test.cpp and lucene_global_index_test.cpp.

IT (test/inte/global_index_test.cpp):

TestFullTextTableFullTextSearch and TestLuceneTableFullTextSearch: two shards, the table
search equals the reader search for every limit, serialized splits read back, the snapshot
selected by scan.snapshot-id (also with scan.mode = from-snapshot, and rejected with
latest-full), scan.tag-name and scan.timestamp-millis, a plan of an earlier snapshot,
reading _INDEX_SCORE, and a bitmap index on the column is ignored. While only the first shard is
indexed, fast returns only the indexed rows, even with global-index.search-mode = full, and
full and detail are rejected.
TestFullTextSearchOverlappingIndexRanges and TestLuceneSearchOverlappingIndexRanges: the
planned shards and search ranges, and the table search equals the reader search.
TestFullTextSearchSkipsRowsDeletedByDeletionVectors and
TestLuceneSearchSkipsRowsDeletedByDeletionVectors: only live rows are ranked, a plan made before
the deletion still ranks its rows, the read falls back to a tag once the snapshot is deleted,
including when an unrelated tag is deleted while the tags are listed, and fails when the snapshot
is missing or unreadable.
TestWriteCommitScanReadFullTextIndexWithPartition and
TestWriteCommitScanReadLuceneIndexWithPartition: partition filters.

API and Format

New public API in include/paimon/table/source/: full_text_search_builder.h, full_text_scan.h,
full_text_read.h and full_text_search_split.h, all marked PAIMON_EXPORT. GlobalIndexer
gains the virtual SupportsFullTextSearch(), which defaults to false, and Options gains
GLOBAL_INDEX_SEARCH_MODE and FULL_TEXT_INDEX_SEARCH_MODE. Full-text search splits have a new
serialized form (magic PFTSPLT1, version 1), specific to Paimon C++. No storage format or
protocol change.

Documentation

Yes. docs/source/user_guide/global_index.rst adds a "Table-Level Full-Text Search" section,
covering local search, the supported search mode, pinning the data scan to the plan snapshot and
distributed search with serialized splits, and docs/source/api/global_index.rst lists the new
classes.

Generative AI tooling

Generated-by: Claude Code 2.1.309 (Claude Opus 5.5)

🤖 Generated with Claude Code

…lution tables Add FullTextSearchBuilder, FullTextScan, FullTextRead and serializable full-text search splits, aligned with Java FullTextSearchBuilder, DataEvolutionFullTextScan and DataEvolutionFullTextRead. - The scan picks its snapshot as Java TimeTravelUtil does: from scan.snapshot-id, scan.tag-name, scan.timestamp or scan.timestamp-millis, or else the latest one. It first checks them against scan.mode with SchemaValidation::ValidateStartupMode, now implemented as Java SchemaValidation does, so that for example scan.snapshot-id is rejected with latest-full. The latest schema of the table is still used, not that of the snapshot. - It applies the partition filter and plans one split per index shard whose indexer supports full-text search. When shards overlap, including multi-field indexes that carry the column as an extra field, each row is ranked by one shard only. A full-text or lucene-fts index whose backend is not built fails the plan instead of being skipped. - The read searches the splits in parallel and keeps the global top-k. With deletion vectors it ranks only the rows live in the snapshot the splits were planned from, loading that snapshot from a tag once it has expired. - Splits carry their snapshot id and use a versioned encoding, so scan and read can run in different processes. The encoding is specific to Paimon C++, as Java serializes its splits with Java serialization. Decoding rejects a null buffer and checks every count, range and binary row layout before reading it, and the read rejects a split whose index files no longer index the column, for example after a column of the same name replaced it. - Add GlobalIndexer::SupportsFullTextSearch(), true for full-text and lucene-fts. - Add the full-text-index.search-mode and global-index.search-mode options, resolved as Java CoreOptions#fullTextIndexSearchMode does. Only the rows that an index covers are searched, so the builder rejects the full and detail modes with NotImplemented. - Share the scored top-k merge, the row id range helpers and executor creation with the existing global index reader and scan. The merge is also compiled into paimon_global_index, whose union reader calls it. Row filters are not supported yet, so splits do not carry the scalar index files that Java uses to pre-filter rows. Columns with a primary-key full-text index are rejected with NotImplemented for now.

lszskye

+1

lxy-9602

+1

Every Lucene global index reader and writer constructed its own cppjieba::Jieba, reloading about 11.6MB of dictionaries each time. A table-level full-text search creates one reader per shard, so every query paid this cost once per shard. After the table-level full-text search tests were added in apache#461, paimon-global-index-test on tsan-x86_64 grew from about 1400s to about 3500s and pushed the job past its 120-minute timeout. Add LuceneUtils::GetJieba(), which loads each dictionary directory once per process and shares a const segmenter with all readers and writers. Segmentation only calls const cppjieba methods, which keep no mutable state and are safe to call concurrently. The cache is keyed by the canonical absolute path, so a relative PAIMON_JIEBA_DICT_DIR resolves against the current working directory instead of reusing dictionaries loaded from another one, and a directory that cannot be resolved returns an IOError instead of aborting in cppjieba. Document how a relative directory is resolved and that dictionary changes take effect only after the process restarts. On tsan-x86_64, this cuts paimon-global-index-test from about 3500s to about 160s, and the job finishes in about an hour instead of timing out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Every Lucene global index reader and writer constructed its own cppjieba::Jieba, reloading about 11.6MB of dictionaries each time. A table-level full-text search creates one reader per shard, so every query paid this cost once per shard. After the table-level full-text search tests were added in apache#461, paimon-global-index-test on tsan-x86_64 grew from about 1400s to about 3500s and pushed the job past its 120-minute timeout. Add LuceneUtils::GetJieba(), which loads each dictionary directory once per process and shares a const segmenter with all readers and writers. Segmentation only calls const cppjieba methods, which keep no mutable state and are safe to call concurrently. The cache is keyed by the canonical absolute path, so a relative PAIMON_JIEBA_DICT_DIR resolves against the current working directory instead of reusing dictionaries loaded from another one, and a directory that cannot be resolved returns an IOError instead of aborting in cppjieba. Document how a relative directory is resolved and that dictionary changes take effect only after the process restarts. On tsan-x86_64, this cuts paimon-global-index-test from about 3500s to about 160s, and the job finishes in about an hour instead of timing out.

SteNicholas force-pushed the PAIMON-404 branch 2 times, most recently from 531e393 to 7424815 Compare October 24, 2026 16:40

lxy-9602 requested a review from lszskye October 26, 2026 01:55