[SteNicholas]

Purpose

Linked issue: close #405

A full-text search in Paimon C++ could not carry a row filter, so a caller had to filter after
ranking and got fewer than limit rows. This adds FullTextSearchBuilder::WithFilter(), aligned
with Java FullTextSearchBuilder#withFilter (apache/paimon#9855): only the matching rows are
ranked, so the result is the top k among them, and they reach the existing index as include row
ids, so BM25 statistics stay those of the full corpus.

• Predicate split: as Java PartitionPredicate#splitPartitionPredicatesAndDataPredicates does,
  conjuncts over partition fields alone are ANDed into the partition filter, and the rest become the
  data filter. A conjunct over both partition and data fields stays in the data filter and also
  restricts the partitions it implies, projected as Java PredicateProjectionConverter does, so
  (p = 1 AND a > 0) OR (p = 2 AND a > 0) also searches only p = 1 OR p = 2. Repeated calls are
  ANDed.
• DataEvolutionFullTextScan: also keeps the index files that index a filter field, and
  attaches to each split the scalar pre-filter files that intersect its shard: files that are not
  full-text indexes, or full-text indexes with extra fields. Unlike in Java, the latter rule holds
  whichever column the index serves, as lucene-fts answers every scalar predicate with all of its
  rows. A scalar index file whose row range is negative, reversed or overflows int64 fails the plan.
• IndexFullTextSearchSplit: version 2 of the encoding appends the scalar index files, as
  version 2 of Java IndexFullTextSearchSplit does; version 1 splits are still read.
• DataEvolutionFullTextRead#matchedRows: resolves the data filter before ranking with the
  scalar indexes of the splits, read per field as one union (GlobalIndexScanImpl:: CreateForScalarFilters and ScanWithCoverage, as Java DataEvolutionGlobalIndexScanner).
  • Rows that no scalar index covers are excluded, as in the fast scalar-index.search-mode. If
    no index can evaluate the filter at all, nothing is searched and a warning is logged.
  • If the indexes return only a superset, the candidates are excluded with a warning by default,
    so the search never returns a non-matching row. With the new
    global-index.filter.refine-from-data option (default false, Java description), they are
    verified against the data instead.
  • Each split searches with include row ids searchRowRanges ∧ liveRows ∧ matchedRows.
• GlobalIndexEvaluatorImpl: reports the fields that decided a result
  (EvaluateWithContributingFields, as Java GlobalIndexEvaluator.Evaluation), so that a superset
  can be told from an exact answer: not every filter field contributed, or there is a Contains,
  EndsWith or Like leaf. Unlike in Java, a field with a conjunct that no index could evaluate
  does not count even if another of its conjuncts was evaluated, as the result is then only a
  superset for that field.
• FilteredRowIdReader: as in Java, reads only the filter columns and _ROW_ID of the
  candidates, pinned to the planned snapshot and with the schema that the search resolved, so a
  column renamed or replaced since then cannot point the filter at another field. Unlike in Java,
  once the planned snapshot has expired, a search of scan.tag-name reads the tag instead, which
  must still refer to the planned snapshot.
• StaticFromTagStartingScanner: reads tags from the branch of the scan, as in Java, instead of
  always from the main branch.
• Options: scalar-index.search-mode falls back on global-index.search-mode, as Java
  CoreOptions#scalarIndexSearchMode does. Only fast is supported so far; a row filter in full
  or detail fails with NotImplemented, as does a filter with a full-text match.

Not part of this change:

• The full and detail scalar search modes ([Feature] Support full-text search modes for unindexed row ranges #406), which decide the rows that no scalar index
  covers by reading their filter columns.
• Row filters on a primary-key full-text index ([Feature] Search primary-key full-text indexes #410), which is not searchable yet.

Tests

UT:

• src/paimon/core/table/source/data_evolution_full_text_read_test.cpp, with fake scalar indexers:
  TestFilterRanksOnlyMatchedRows, TestFilterExcludesRowsThatIndexesCannotDecide (no scalar
  file, a declining or multi-field one, a partly indexed conjunction, Contains/EndsWith/Like,
  rows that no scalar file covers), TestFilterUnionsIndexesOfDifferentTypes, and filter validation in
  TestBuilderValidatesSettings (unknown field, _ROW_ID, wrong type, full-text match, a scalar
  search mode other than fast, null filter).
• src/paimon/core/table/source/data_evolution_full_text_scan_test.cpp: TestScalarPreFilterFiles
  and TestIntersectingFiles, including invalid row ranges.
• src/paimon/core/table/source/full_text_search_split_test.cpp: TestSerializeRoundTrip (the
  version 2 bytes are pinned and version 1 bytes still read), TestSerializeScalarIndexFiles, and
  invalid scalar index files in TestDeserializeRejectsInvalidBytes,
  TestDeserializeRejectsCorruptEntries and TestCreateValidatesArguments.
• src/paimon/core/table/source/filtered_row_id_reader_test.cpp:
  TestExactWhenEveryFilterFieldContributed and TestCandidateOnlyLeavesAreNotExact.
• src/paimon/core/global_index/global_index_evaluator_impl_test.cpp: ReportsContributingFields,
  AndWithEmptyResultShortCircuits and CollectFieldIds.
• src/paimon/core/core_options_test.cpp: TestParseIndexSearchMode for the full-text and scalar
  families, and global-index.filter.refine-from-data in TestDefaultValue and TestFromMap.

IT (test/inte/global_index_test.cpp), aligned with Java NativeFullTextRowFilterTest and
FullTextSearchBuilderTest:

• TestFullTextTableFullTextSearchWithFilter and TestLuceneTableFullTextSearchWithFilter:
  equality, range, IN, NotEqual, OR across fields, no and all matching rows, equality on a
  BTree of the text column, accumulated filters, the top row among matching rows with limit 1,
  a column without an index or with only a full-text index, a partly indexed conjunction and
  Contains/EndsWith/Like with refine on and off, scalar index files attached only for filtered
  columns, serialized splits, and refine with scan.snapshot-id, scan.tag-name and
  scan.timestamp-millis.
• TestFullTextTableFullTextSearchWithPartitionAndDataFilter and
  TestLuceneTableFullTextSearchWithPartitionAndDataFilter: partition and data predicates combined,
  with SetPartitionFilter() and refine, and the partitions that a mixed conjunct implies.
• TestFullTextSearchSkipsRowsDeletedByDeletionVectors and
  TestLuceneSearchSkipsRowsDeletedByDeletionVectors: a filter with deletion vectors, a scalar index
  file per shard, and refine across shards.
• TestFilteredRowIdReader: deletion vectors, a pinned older snapshot, a tag moved to another
  snapshot, the tag fallback once the snapshot has expired (also with an empty plan), a branch tag,
  and a column renamed or replaced after planning.

API and Format

New public API: FullTextSearchBuilder::WithFilter(), and Options::SCALAR_INDEX_SEARCH_MODE and
Options::GLOBAL_INDEX_FILTER_REFINE_FROM_DATA. The serialized form of full-text search splits,
which is specific to Paimon C++, moves to version 2 and still reads version 1. No storage format or
protocol change.

Documentation

Yes. docs/source/user_guide/global_index.rst adds a "Row Filters" section covering the predicate
split, implied partitions, the fast mode, excluded candidates and
global-index.filter.refine-from-data, and the docs of FullTextSearchBuilder, FullTextScan,
FullTextRead and the new options describe the filter.

Generative AI tooling

Generated-by: Claude Code 2.1.311 (Claude Opus 5.5)

🤖 Generated with Claude Code