[Feature] Align Full Text Index with Java Paimon #399

Search before asking

• I searched in the issues and found nothing similar.

Motivation

The full-text global index in Paimon C++ (src/paimon/global_index/tantivy/, crates/tantivy_ffi/) follows the removed Java paimon-tantivy module; the Java cross-read fixtures in test/test_data/java_tantivy_fixtures were generated on 2026-04-23. Since then Java Paimon has added configurable analyzers (apache/paimon#8074) and a JSON DSL query (apache/paimon#8308) to that module, replaced it with the standalone native engine paimon-full-text-index 0.1.0 from apache/paimon-full-text (apache/paimon#8463, apache/paimon#8467, apache/paimon#8798), and built table-level and primary-key full-text search on top of it. As a result, full-text index files are no longer interoperable between Java and C++ in either direction, and most search features are missing in C++.

Index file format and engine

• The paimon-ftindex 0.1.0 reader rejects both test/test_data/cpp_tantivy_fixtures/english_default.archive and test/test_data/java_tantivy_fixtures/english_simple.archive with invalid storage format: bad magic.
• ArchiveLayout::Parse reads the PFTI magic as the file count, so C++ cannot read Java archives either. Because no indexer is registered as full-text, GlobalIndexScanImpl skips Java data-evolution full-text indexes silently, and GlobalIndexWriteTask rejects full-text as an unknown index type.

Analyzer options

• Java supports full-text.tokenizer (default, simple, whitespace, raw, ngram, jieba), ngram.min-gram, ngram.max-gram, ngram.prefix-only, jieba.search-mode, jieba.ordinal-position, lower-case, max-token-length, ascii-folding, stem, language, remove-stop-words, stop-words and with-position.
• C++ supports tantivy.write.tokenizer (default, paimon_jieba, whitespace, raw, en_stem), jieba.tokenize-mode and write.omit-term-freq-and-position.
• The default tokenizer behaves differently. Java applies lower-casing, English stemming, stop-word removal and ASCII folding, so a run query matches running. C++ uses tantivy's SimpleTokenizer with lower-casing only.

Query model (include/paimon/predicate/full_text_search.h)

• Java takes a JSON DSL query: match (operator, boost, fuzziness, max_expansions, prefix_length), match_phrase (slop), boolean (must/should/must_not), multi_match and boost (negative_boost). The limit is required and positive, and results are always scored.
• C++ takes a SearchType enum (MATCH_ALL, MATCH_ANY, PHRASE, PREFIX, WILDCARD), an optional limit, with_score (default false), min_score and a global row-id pre_filter.

Data-evolution table search

C++ only exposes the reader primitive (GlobalIndexScan::CreateReaders -> VisitFullTextSearch -> SetGlobalIndexResult). Compared with Java:

• UnionGlobalIndexReader::VisitFullTextSearch ORs shard results without a final top-k, so limit = k can return up to shards x k rows. Java merges per-split results and applies a final top-k (ScoredGlobalIndexResult#merge and #topK, [core] Stabilize scored global index top-k tie-breaking paimon#8301, [core][spark] Flatten scored global index result unions paimon#8809), and each split only requests limit candidates ([core] Request only top-k candidates from data-evolution full-text indexes paimon#9955).
• Deleted rows are not excluded before ranking (fix: filter deleted rows from global index search paimon#8459, [core] Support full-text indexes on DV tables paimon#9031), so they can take top-k slots and the query returns fewer than limit rows.
• OffsetGlobalIndexReader::VisitFullTextSearch shifts every pre_filter row id by the shard offset without clipping it to the shard range. Java clips it with FullTextSearch#offsetRange (fix: filter deleted rows from global index search paimon#8459).
• Missing features:
  • FullTextSearchBuilder, FullTextScan, FullTextRead and serializable search splits.
  • Row filters via withFilter, resolved into include row ids before ranking, and global-index.filter.refine-from-data ([core][spark][docs] Support row filters in full-text search paimon#9855).
  • full-text-index.search-mode and global-index.search-mode, including full/detail modes that search unindexed row ranges through a temporary index ([core] Add global index search mode paimon#8296, [core] Support full mode for full-text search paimon#8316, [core][python] Fix index-specific search mode defaults paimon#8844).
  • Range selection across overlapping and multi-field indexes ([global-index] Fix full-text range selection and scalar-filter coverage for multi-field indexes paimon#8547).
  • Hybrid search with rrf, weighted_score and mrr rankers ([core][spark][python] Support hybrid search paimon#8271, [core] Stabilize hybrid RRF tie-breaking paimon#8288, [global-index] Normalize hybrid weighted score fusion paimon#8294, [global-index] Optimize hybrid ranker top-k paimon#8348, [vector] Add MRR hybrid search ranker paimon#8351). It also needs a table-level vector search builder, which C++ does not have yet.
  • supportsFullTextSearch() on indexer factories ([core] Align full-text search with JSON DSL paimon#8308).

Primary-key full-text index (pk-full-text.index.columns)

Java: apache/paimon#8649, apache/paimon#8651, apache/paimon#8652, apache/paimon#8659, apache/paimon#8672, apache/paimon#8844, apache/paimon#8992, apache/paimon#9184.

• PrimaryKeyIndexDefinitions creates the FULL_TEXT definition with an empty options map. Java merges table-level full-text.* options with fields.<column>.pk-full-text.index.options and rejects conflicting values.
• Schema validation is missing: SchemaValidation::ValidatePrimaryKeyBTreeIndexes returns early when pk-btree.index.columns is empty, so full-text-only tables are not checked at all. Java requires exactly one column, a primary-key table, deletion vectors (unless the merge engine is first-row) with deletion-vectors.merge-on-read = false, fixed or postpone bucket mode, no pk-clustering-override, a CHAR/VARCHAR column, valid options JSON, and at most one primary-key index per column.
• The write side is missing: per-level archive building (PkFullTextIndexFile), bucket index state, and BucketedFullTextIndexMaintainer with atomic archive replacement on compaction. BucketedPrimaryKeyIndexMaintainer only keeps BTree definitions.
• The read side is missing: PrimaryKeyFullTextScan, PrimaryKeyFullTextSearchSplit, PrimaryKeyFullTextBucketSearch (deletion vectors, global top-k, parallel search) and PrimaryKeyFullTextRead. PrimaryKeySortedIndexScan rejects full-text search, and indexed scores are not propagated through the primary-key positional read path.
• When C++ compacts a table that has Java-written primary-key full-text archives, the archives are neither rebuilt nor retired.

Solution

The work is split into the following sub-issues. Each can be delivered as a separate PR; dependencies are listed in each sub-issue.

1. Index format and engine — [Feature] Replace the tantivy FFI with the paimon-full-text-index native engine #400: replace crates/tantivy_ffi with the C API of paimon-full-text-index (apache/paimon-full-text v0.1.0), register the indexer as full-text, use the full-text. option and file prefixes, store the options as flat JSON metadata, and add Java <-> C++ cross-read tests with fixtures generated by the Java paimon-full-text module.
2. Query model — [Feature] Align FullTextSearch with the Java JSON DSL query model #401: align FullTextSearch with the JSON DSL query, a required positive limit and scored results, and decide whether SearchType stays as a compatibility layer for the C++-only lucene-fts backend.
3. Search correctness — [Bug] Full-text search over multiple index shards returns up to shards × limit rows #402 (final top-k after the union, pre_filter clipping to the shard range) and [Bug] Deleted rows take full-text top-k slots in data-evolution tables #403 (exclude deleted rows before ranking).
4. Table-level full-text search — [Feature] Support table-level full-text search for data-evolution tables #404 (builder, scan, read, splits and range selection), [Feature] Support row filters in full-text search #405 (row filters), [Feature] Support full-text search modes for unindexed row ranges #406 (search modes) and [Feature] Support hybrid search with rrf, weighted_score and mrr rankers #407 (hybrid search rankers).
5. Primary-key full-text index — [Feature] Resolve primary-key full-text index options and validate schemas #408 (options merging and schema validation), [Feature] Maintain primary-key full-text index archives on write and compaction #409 (write-side maintenance) and [Feature] Search primary-key full-text indexes #410 (read-side bucket search).

Anything else?

The format findings were checked with the paimon-ftindex 0.1.0 wheel from PyPI, which wraps the same native engine as the Java paimon-full-text module. An index written by it has the PFTIDX01 header described above, and its reader rejects both C++ fixture archives. The other items come from comparing the Java paimon-full-text and paimon-core sources with the C++ sources.

Are you willing to submit a PR?

• I'm willing to submit a PR!