docs: fix drifted statements, feature lists and code examples

README.md

@@ -31,12 +31,12 @@ Background and documentation are available at [paimon.apache.org](https://paimon
 Paimon C++ currently provides:
 
 - **Write**: append table and primary key table write support with compaction.
-- **Commit**: append table commit support for simple append-only tables.
-- **Scan**: batch and stream scan for append tables and primary key tables without changelog.
+- **Commit**: append table and primary key table commit support. Direct file-system commits work on non-object-store paths; object-store paths require a catalog-mediated commit.
+- **Scan**: batch and stream scan for append tables and primary key tables, with changelog consumption when a changelog producer is configured.
 - **Read**: append table read, primary key table read with deletion vector, and primary key table merge-on-read.
 - **Arrow integration**: batch read and write interfaces based on the [Arrow Columnar In-Memory Format](https://arrow.apache.org).
 - **File systems**: file system abstraction with built-in local, Jindo, OSS, and S3 file system support.
-- **File formats**: file format abstraction with built-in ORC, Parquet, and Avro support.
+- **File formats**: file format abstraction with built-in Parquet, Blob, ORC, and Avro support, plus optional Lance and Mosaic formats.
 - **Runtime utilities**: memory pool and thread pool abstractions with default implementations.
 - **AI-Oriented Features**: supports RowTracking and DataEvolution mode and provides Global Index
   capabilities including B-tree index, DiskANN-based vector search with Lumina, and Lucene-based

docs/code-style.md

@@ -404,6 +404,11 @@ Before writing new helper code, check whether the project already provides what
 | `UniqueTestDirectory` | RAII helper that creates a unique temp directory and cleans it up on destruction |
 | `TimezoneGuard` | RAII guard that sets `TZ` environment variable and restores it on destruction |
 | `IOExceptionHelper` | Inject I/O errors via `IOHook` for fault-injection testing |
+| `CountingRoutingCache` | `Cache` routing by `CacheKind` to a per-kind `LruCache`, counting gets and supplier calls overall and per kind |
+| `DeletionVectorTestHelper` | Express row-level deletes the way an external engine does: write a deletion vector index file and commit it as an index-only commit message |
+| `GatedAsyncInputStream` | `InputStream` wrapper holding `ReadAsync` callbacks until `ReleaseAll()`, so a test can observe a cache while its fetches are in flight |
+| `BuildTestSnapshot()` | Build a `Snapshot` from an id and optional uuid, filling in every other field |
+| `VariantTestData` | Build variant test data, including an Arrow `StructArray` batch |
 | `testharness.h` | Test macros: `ASSERT_OK`, `ASSERT_NOK`, `ASSERT_NOK_WITH_MSG`, `ASSERT_RAISES` |
 
 ### Mock Objects (`src/paimon/testing/mock/`)
@@ -417,6 +422,13 @@ Before writing new helper code, check whether the project already provides what
 | `MockIndexPathFactory` | Mock index path factory |
 | `MockKeyValueDataFileRecordReader` | Mock key-value data file reader holding in-memory data |
 | `MockStatsExtractor` | Mock statistics extractor |
+| `MockInputStream` / `MockOutputStream` | No-op I/O doubles: reads and writes report zero bytes and `ReadAsync` completes inline |
+| `MockVersionManagedCatalog` | Mock `Catalog` that also implements `VersionManagedCatalog` and records every commit call |
+| `MockGlobalIndexReader` | Mock `GlobalIndexReader`, constructed with or without search support |
+| `MockSearchFileIndex` | File index backend with configured matches in search order; applies pre-filters before limits and returns vector scores aligned with ascending file-local row positions |
+| `MockSearchFileIndexer` / `MockSearchFileIndexReader` | Mock `FileIndexer` and `FileIndexReader` serving those configured scored rows |
+| `MockFullTextSearchFileIndexFactory` / `MockVectorSearchFileIndexFactory` | Mock `FileIndexerFactory` for the full-text and vector search index types |
+| `OptionsCheckingMockSearchFileIndexFactory` | Reuses the mock search backend while requiring the exact options supplied to its factory |
 
 ---
 
@@ -453,6 +465,8 @@ This installs Git hooks that automatically run on every commit:
 | `trailing-whitespace` | Trailing whitespace |
 | `end-of-file-fixer` | Missing newline at end of file |
 | `check-added-large-files` | Files > 5 MB |
+| `check-yaml` | YAML syntax |
+| `sphinx-lint` (alias `docs`) | reStructuredText syntax under `docs/source/` |
 
 To run all hooks on all files manually:
 
@@ -464,10 +478,12 @@ pre-commit run --all-files
 
 Static analysis is configured in `.clang-tidy`. Key enabled check groups:
 
-- `bugprone-*` — Common bug patterns
-- `google-*` — Google style checks
-- `modernize-*` — C++ modernization suggestions
-- `clang-analyzer-*` — Clang static analyzer
+- `bugprone-*` — Common bug patterns. These are listed one check at a time rather than enabled as a wildcard, so a new upstream `bugprone` check never turns itself on.
+- `misc-misleading-identifier`, `misc-homoglyph` — The only two `misc` checks enabled; the rest of `misc-*` is off.
+- `google-*` — Google style checks, except `google-default-arguments`.
+- `modernize-*` — C++ modernization suggestions, except `modernize-return-braced-init-list`, `modernize-avoid-c-arrays`, `modernize-use-trailing-return-type`, `modernize-use-nodiscard` and `modernize-pass-by-value`.
+- `clang-analyzer-*` — Clang static analyzer, except the `clang-analyzer-alpha*` checkers and `clang-analyzer-cplusplus.NewDeleteLeaks`.
+- `clang-diagnostic-*` — Compiler diagnostics, except `clang-diagnostic-global-constructors` and `clang-diagnostic-sign-compare`.
 
 ### Building & Testing
 

docs/source/api/catalog.rst

@@ -30,3 +30,12 @@ Interface
 .. doxygenclass:: paimon::Identifier
    :members:
    :undoc-members:
+
+.. doxygenclass:: paimon::Database
+   :members:
+
+.. doxygenclass:: paimon::Table
+   :members:
+
+.. doxygenstruct:: paimon::CatalogOptions
+   :members:

docs/source/build_system.rst

@@ -102,6 +102,8 @@ Paimon provides a set of built-in optional plugins that you can link to as neede
   - ``Paimon::paimon_avro_file_format_shared``
   - ``Paimon::paimon_blob_file_format_shared``
   - ``Paimon::paimon_lance_file_format_shared``
+  - ``Paimon::paimon_mosaic_file_format_shared``
+  - ``Paimon::paimon_vortex_file_format_shared``
 
 - File system plugins:
 
@@ -113,8 +115,17 @@ Paimon provides a set of built-in optional plugins that you can link to as neede
 - Index plugins:
 
   - ``Paimon::paimon_file_index_shared``
+  - ``Paimon::paimon_global_index_shared``
   - ``Paimon::paimon_lumina_index_shared``
   - ``Paimon::paimon_lucene_index_shared``
+  - ``Paimon::paimon_full_text_index_shared``
+
+.. note::
+
+  A plugin target only exists when its component is enabled. ``parquet``,
+  ``blob``, ``local``, ``file_index`` and ``global_index`` are always built; the
+  others require the matching ``PAIMON_ENABLE_*`` flag described in
+  :ref:`cpp_build_optional_components`.
 
 .. note::
 

docs/source/building.rst

@@ -173,12 +173,20 @@ Optional Components
 
 By default, the C++ build system creates a fairly minimal build. We have
 several optional system components which you can opt into building by passing
-boolean flags to ``cmake``.
+boolean flags to ``cmake``. ORC and Avro are the exceptions: both already default
+to ``ON``, while every other component below defaults to ``OFF``.
 
 * ``-DPAIMON_ENABLE_ORC=ON``: Paimon integration with Apache ORC
 * ``-DPAIMON_ENABLE_AVRO=ON``: Apache Avro libraries and Paimon integration
 * ``-DPAIMON_ENABLE_LANCE=ON``: Lance 0.39 file format support through a Rust FFI; requires
   Rust 1.88 or newer, cbindgen, and protoc 3.12 or newer
+* ``-DPAIMON_ENABLE_MOSAIC=ON``: Apache Paimon Mosaic file format support through a Rust FFI;
+  requires cargo on ``PATH``
+* ``-DPAIMON_ENABLE_VORTEX=ON``: Vortex file format support through a Rust FFI; requires cargo,
+  the stable Rust toolchain (Vortex pins ``rust-version`` 1.95 or newer and is built with
+  ``RUSTUP_TOOLCHAIN=stable``), and a ``flatc`` whose version matches the flatbuffers crate that
+  Vortex uses. Distribution ``flatc`` packages are typically too old; set ``FLATC`` to the path of
+  a suitable binary when it is not on ``PATH``.
 * ``-DPAIMON_ENABLE_JINDO=ON``: Support for Alibaba Jindo filesystems
 * ``-DPAIMON_ENABLE_OSS=ON``: Support for Alibaba Cloud OSS through OSS SDK V2
 * ``-DPAIMON_ENABLE_S3=ON``: Support for Amazon S3-compatible filesystems
@@ -278,6 +286,8 @@ For development builds, you will often want to enable additional targets in
 enable to exercise your changes, using the following ``cmake`` options.
 
 * ``-DPAIMON_BUILD_TESTS=ON``: Build executable unit tests.
+* ``-DPAIMON_ENABLE_NETWORK_TESTS=ON``: Enable tests that access real remote services over the
+  network. These are disabled by default.
 
 Optional Checks
 ~~~~~~~~~~~~~~~

docs/source/user_guide/catalog.rst

@@ -158,6 +158,13 @@ offsets when a real-time writer is created:
    PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::FileStoreWrite> writer,
                           paimon::FileStoreWrite::Create(std::move(write_context)));
 
+The ``Table`` handle is an equivalent entry point: ``table->NewCommit()``,
+``table->NewWrite()``, ``table->NewRead()`` and ``table->NewScan()`` each take the
+same finished context and return the same operator as ``FileStoreCommit::Create()``,
+``FileStoreWrite::Create()``, ``TableRead::Create()`` and ``TableScan::Create()``.
+Pick whichever reads better; the context still carries the catalog when one is
+needed. See the :ref:`Table API <cpp-api-catalog>`.
+
 For real-time writes, also set ``WithStreamingMode(true)`` and
 ``WithRealtimeContext(...)`` with ``realtime.enabled=true``. After
 ``CommitWithProgress`` succeeds, pass its returned snapshot id to
@@ -431,12 +438,22 @@ file system of your own when the credentials expire.
 
    class MyCredentialProvider : public paimon::CredentialProvider {
     public:
+       // The factory hands over the path and the file system options of the accesses to
+       // authenticate, which is where you read your own configuration from.
+       MyCredentialProvider(std::string path, std::map<std::string, std::string> options)
+           : path_(std::move(path)), options_(std::move(options)) {}
+
        // Called as each access is signed; reload the credentials before they expire.
        paimon::Result<std::map<std::string, std::string>> GetCredentials() const override {
+           // Request fresh credentials for path_ from the token service in options_.
            return std::map<std::string, std::string>{
                {"fs.oss.accessKeyId", ...}, {"fs.oss.accessKeySecret", ...},
                {"fs.oss.securityToken", ...}};
        }
+
+    private:
+       std::string path_;
+       std::map<std::string, std::string> options_;
    };
 
    class MyCredentialProviderFactory : public paimon::CredentialProviderFactory {

docs/source/user_guide/compaction.rst

@@ -26,7 +26,10 @@ compaction is therefore a trade-off between write throughput and read efficiency
 .. note::
    - There can only be one job working on the same partition's compaction,
      otherwise it will cause conflicts.
-   - Paimon C++ does not support producing changelog for now.
+   - Changelog is produced according to the ``changelog-producer`` table option
+     (``input``, ``full-compaction`` or ``lookup``): the ``full-compaction`` and
+     ``lookup`` producers emit their changelog files from compaction, and the
+     files are committed with their snapshot.
    - Automatic compaction is disabled when ``write-only`` is set to ``true``,
      or when an append-only table uses unaware-bucket mode (``bucket = -1``).
    - For a complete list of compaction-related configurations, see the
@@ -78,8 +81,8 @@ Full Compaction
 Full compaction rewrites all eligible files in the bucket. During full
 compaction:
 
-- Files whose size is already at or above ``compaction.file-size`` (and have no
-  associated deletion vectors) are skipped to avoid unnecessary rewrites.
+- Files whose size is already at or above the compaction file-size threshold (and
+  have no associated deletion vectors) are skipped to avoid unnecessary rewrites.
 - When deletion vectors are enabled, all files are always eligible for
   compaction regardless of size, because deletion vectors must be applied.
 - When ``compaction.force-rewrite-all-files`` is ``true``, all files are
@@ -88,10 +91,17 @@ compaction:
   small files exceeds the number of large files and the total file count is at
   least 3.
 
-After compaction, if the last output file is still smaller than
-``compaction.file-size``, it is placed back into the compaction queue for future
+After compaction, if the last output file is still smaller than the compaction
+file-size threshold, it is placed back into the compaction queue for future
 merging.
 
+The compaction file-size threshold is not a separate option. It is derived from
+``target-file-size`` as ``target-file-size / 10 * 7``, that is 70% of the target
+size, so that a file already close to the target is not compacted twice.
+``target-file-size`` defaults to 256 MiB for append-only tables and 128 MiB for
+primary-key tables, which makes the derived threshold about 179 MiB and 89 MiB
+respectively. Adjust ``target-file-size`` to move the threshold.
+
 Dictionary Passthrough
 ~~~~~~~~~~~~~~~~~~~~~~
 An append-only compaction rewrite copies rows into the new file without
@@ -126,7 +136,7 @@ dictionary-encodes it the same way; this option only selects ``STRING`` columns.
 A file with a serialized Arrow dictionary schema can independently produce a
 ``BINARY`` dictionary, which the value accessors also support. The rewrite also
 overrides the option back to ``false`` when the table writes a format other than Parquet, when
-``parquet.enable-dictionary`` is ``false`` because the writer would only expand
+``parquet.enable.dictionary`` is ``false`` because the writer would only expand
 the values again, or when variant/map shredding is configured because those
 writers reshape each batch against a fixed physical schema. Setting the option
 therefore never makes a rewrite fail; at worst it has no effect.

docs/source/user_guide/manifest_entry_cache.rst

@@ -86,6 +86,14 @@ under
            .AddOption(Options::SCAN_MANIFEST_ENTRY_CACHE_MAX_SNAPSHOTS, "3")
            .Finish());
 
+.. note::
+
+   ``LruCache`` is the built-in weight-bounded implementation, but it is declared in
+   ``src/paimon/common/io/cache/lru_cache.h`` and only ``include/`` is installed, so it is
+   not reachable from an installed Paimon. ``WithCache`` takes the public ``Cache``
+   interface of ``include/paimon/cache/cache.h``; pass your own implementation of it when
+   you build against the installed package.
+
 Cache entries are scoped by table path, branch, and bucket, so they can be
 reused across newly created ``TableScan`` and ``FileStoreScan`` instances as
 long as they share the same cache object and scan the same bucket.

docs/source/user_guide/primary_key_global_index.rst

@@ -84,10 +84,11 @@ Current scope
   definitions are recognized for validation, but their evaluation conservatively falls
   back to a normal scan until their dedicated payload readers are supported.
 - The read path targets the Java release-2.0.0 layout and scan semantics (source metadata
-  v1, ``GlobalIndexMeta`` with ``_SOURCE_META``, commit message v12). Source-file names
-  currently use the existing C++ length-prefixed UTF-8 streams; ASCII and non-null BMP
-  names are compatible with Java ``writeUTF``, while complete modified UTF-8 support for
-  supplementary code points will be handled by a shared stream-level change.
+  v1, ``GlobalIndexMeta`` with ``_SOURCE_META``). Commit messages are written at version 13
+  and remain readable down to version 3. Source-file names currently use the existing C++
+  length-prefixed UTF-8 streams; ASCII and non-null BMP names are compatible with Java
+  ``writeUTF``, while complete modified UTF-8 support for supplementary code points will be
+  handled by a shared stream-level change.
 - Automatic maintenance is synchronous during prepare-commit. Java's asynchronous build
   scheduling, retries, fairness metrics, and manual rebuild actions are not part of the C++
   API. Realtime and postpone-bucket writers do not build source-backed payloads.

docs/source/user_guide/primary_key_table.rst

@@ -38,7 +38,9 @@ efficient querying.
 Each bucket directory contains an LSM tree and its changelog files.
 
 .. note::
-   Changelog is not supported yet for Paimon C++ primary key table write.
+   Primary key table writes produce changelog according to the
+   ``changelog-producer`` table option, and the changelog files are committed
+   with their snapshot.
 
 The range for a bucket is determined by the hash value of one or more columns in
 the records. Users can specify bucketing columns by providing the bucket-key option.

docs/source/user_guide/system_tables.rst

@@ -26,6 +26,59 @@ Branch-qualified paths are also supported. For example,
 ``/warehouse/db.db/orders$branch_audit$files`` reads the files system table from
 the ``audit`` branch.
 
+Table-Level System Tables
+-------------------------
+
+These suffixes are recognized on a data table path. Matching is
+case-insensitive, and an unrecognized suffix fails with
+``unsupported system table: <name>`` instead of falling back to the base table.
+
+* ``$options``: the latest base table options as key/value rows.
+* ``$snapshots``: snapshot commit history.
+* ``$schemas``: schema evolution history.
+* ``$tags``: tags and the snapshots they reference.
+* ``$branches``: table branches, including ``main``.
+* ``$consumers``: persisted streaming consumer offsets.
+* ``$manifests``: data manifest metadata in the latest snapshot.
+* ``$files``: data file metadata in the latest snapshot.
+* ``$audit_log``: row-level changelog records with rowkind.
+* ``$binlog``: changelog records with list-wrapped data columns.
+* ``$ro``: read-optimized data, described in the next section.
+
+Global System Tables
+--------------------
+
+Four system tables are not bound to a single data table. They live under the
+reserved ``sys`` database, and three of them enumerate across all databases in
+the warehouse:
+
+* ``sys.catalog_options``: catalog-level configuration as key/value rows.
+* ``sys.all_table_options``: all table options as
+  ``(database_name, table_name, key, value)`` rows.
+* ``sys.tables``: metadata for all tables, including record counts and file
+  statistics.
+* ``sys.partitions``: partition-level file statistics for all tables.
+
+``sys.all_table_options``, ``sys.tables`` and ``sys.partitions`` are always
+available. ``sys.catalog_options`` is disabled by default and has to be turned
+on with the catalog option ``catalog-options-table.enabled``; a disabled one is
+reported as not existing.
+
+Those three enumerate databases and tables, so they need a catalog: loading a
+bare ``<warehouse>/sys/<name>`` path fails with ``global system table requires
+catalog context: <name>``. ``sys.catalog_options`` enumerates nothing, and is
+the one a bare path serves.
+
+.. note::
+
+   No public read path serves those three yet. A catalog lists them and
+   returns their schema, but does not hand back a readable table:
+   ``Catalog::GetTable(Identifier("sys", "tables"))`` is rejected by a
+   file-system catalog with ``do not support checking TableSchemaExists for
+   system table``, and a REST catalog returns one whose path is left unset,
+   which ``TableScan`` and ``TableRead`` cannot dispatch to the global system
+   table loader.
+
 Read-Optimized System Table
 ---------------------------