From 4ad4218297cbb9c41d6d06ad01b333e5e592940c Mon Sep 17 00:00:00 2001 From: "yonghao.fyh" Date: Mon, 26 Oct 2026 17:55:27 +0800 Subject: [PATCH 1/3] docs: fix drifted option names, version numbers and feature lists --- README.md | 4 +- docs/code-style.md | 24 +++++++-- docs/source/api/catalog.rst | 9 ++++ docs/source/build_system.rst | 11 ++++ docs/source/building.rst | 12 ++++- docs/source/user_guide/catalog.rst | 17 ++++++ docs/source/user_guide/compaction.rst | 17 ++++-- .../user_guide/manifest_entry_cache.rst | 8 +++ .../user_guide/primary_key_global_index.rst | 9 ++-- docs/source/user_guide/system_tables.rst | 53 +++++++++++++++++++ docs/source/user_guide/write.rst | 14 +++-- .../map_shared_shredding_schema_utils.h | 12 +++-- include/paimon/data/variant.h | 10 ++-- include/paimon/read_context.h | 5 +- src/paimon/common/data/blob_defs.h | 2 +- 15 files changed, 176 insertions(+), 31 deletions(-) diff --git a/README.md b/README.md index da928e713..e6d4e5a5e 100644 --- a/README.md +++ b/README.md @@ -31,12 +31,12 @@ Background and documentation are available at [paimon.apache.org](https://paimon Paimon C++ currently provides: - **Write**: append table and primary key table write support with compaction. -- **Commit**: append table commit support for simple append-only tables. +- **Commit**: append table and primary key table commit support. Direct file-system commits work on non-object-store paths; object-store paths require a catalog-mediated commit. - **Scan**: batch and stream scan for append tables and primary key tables without changelog. - **Read**: append table read, primary key table read with deletion vector, and primary key table merge-on-read. - **Arrow integration**: batch read and write interfaces based on the [Arrow Columnar In-Memory Format](https://arrow.apache.org). - **File systems**: file system abstraction with built-in local, Jindo, OSS, and S3 file system support. -- **File formats**: file format abstraction with built-in ORC, Parquet, and Avro support. +- **File formats**: file format abstraction with built-in Parquet, Blob, ORC, and Avro support, plus optional Lance and Mosaic formats. - **Runtime utilities**: memory pool and thread pool abstractions with default implementations. - **AI-Oriented Features**: supports RowTracking and DataEvolution mode and provides Global Index capabilities including B-tree index, DiskANN-based vector search with Lumina, and Lucene-based diff --git a/docs/code-style.md b/docs/code-style.md index cb8c648e8..3772f0f33 100644 --- a/docs/code-style.md +++ b/docs/code-style.md @@ -404,6 +404,11 @@ Before writing new helper code, check whether the project already provides what | `UniqueTestDirectory` | RAII helper that creates a unique temp directory and cleans it up on destruction | | `TimezoneGuard` | RAII guard that sets `TZ` environment variable and restores it on destruction | | `IOExceptionHelper` | Inject I/O errors via `IOHook` for fault-injection testing | +| `CountingRoutingCache` | `Cache` routing by `CacheKind` to a per-kind `LruCache`, counting gets and supplier calls overall and per kind | +| `DeletionVectorTestHelper` | Express row-level deletes the way an external engine does: write a deletion vector index file and commit it as an index-only commit message | +| `GatedAsyncInputStream` | `InputStream` wrapper holding `ReadAsync` callbacks until `ReleaseAll()`, so a test can observe a cache while its fetches are in flight | +| `BuildTestSnapshot()` | Build a `Snapshot` from an id and optional uuid, filling in every other field | +| `VariantTestData` | Build variant test data, including an Arrow `StructArray` batch | | `testharness.h` | Test macros: `ASSERT_OK`, `ASSERT_NOK`, `ASSERT_NOK_WITH_MSG`, `ASSERT_RAISES` | ### Mock Objects (`src/paimon/testing/mock/`) @@ -417,6 +422,13 @@ Before writing new helper code, check whether the project already provides what | `MockIndexPathFactory` | Mock index path factory | | `MockKeyValueDataFileRecordReader` | Mock key-value data file reader holding in-memory data | | `MockStatsExtractor` | Mock statistics extractor | +| `MockInputStream` / `MockOutputStream` | No-op I/O doubles: reads and writes report zero bytes and `ReadAsync` completes inline | +| `MockVersionManagedCatalog` | Mock `Catalog` that also implements `VersionManagedCatalog` and records every commit call | +| `MockGlobalIndexReader` | Mock `GlobalIndexReader`, constructed with or without search support | +| `MockSearchFileIndex` | File index backend with configured matches in search order; applies pre-filters before limits and returns vector scores aligned with ascending file-local row positions | +| `MockSearchFileIndexer` / `MockSearchFileIndexReader` | Mock `FileIndexer` and `FileIndexReader` serving those configured scored rows | +| `MockFullTextSearchFileIndexFactory` / `MockVectorSearchFileIndexFactory` | Mock `FileIndexerFactory` for the full-text and vector search index types | +| `OptionsCheckingMockSearchFileIndexFactory` | Reuses the mock search backend while requiring the exact options supplied to its factory | --- @@ -453,6 +465,8 @@ This installs Git hooks that automatically run on every commit: | `trailing-whitespace` | Trailing whitespace | | `end-of-file-fixer` | Missing newline at end of file | | `check-added-large-files` | Files > 5 MB | +| `check-yaml` | YAML syntax | +| `sphinx-lint` (alias `docs`) | reStructuredText syntax under `docs/source/` | To run all hooks on all files manually: @@ -464,10 +478,12 @@ pre-commit run --all-files Static analysis is configured in `.clang-tidy`. Key enabled check groups: -- `bugprone-*` — Common bug patterns -- `google-*` — Google style checks -- `modernize-*` — C++ modernization suggestions -- `clang-analyzer-*` — Clang static analyzer +- `bugprone-*` — Common bug patterns. These are listed one check at a time rather than enabled as a wildcard, so a new upstream `bugprone` check never turns itself on. +- `misc-misleading-identifier`, `misc-homoglyph` — The only two `misc` checks enabled; the rest of `misc-*` is off. +- `google-*` — Google style checks, except `google-default-arguments`. +- `modernize-*` — C++ modernization suggestions, except `modernize-return-braced-init-list`, `modernize-avoid-c-arrays`, `modernize-use-trailing-return-type`, `modernize-use-nodiscard` and `modernize-pass-by-value`. +- `clang-analyzer-*` — Clang static analyzer, except the `clang-analyzer-alpha*` checkers and `clang-analyzer-cplusplus.NewDeleteLeaks`. +- `clang-diagnostic-*` — Compiler diagnostics, except `clang-diagnostic-global-constructors` and `clang-diagnostic-sign-compare`. ### Building & Testing diff --git a/docs/source/api/catalog.rst b/docs/source/api/catalog.rst index c3956d32e..df247f4a8 100644 --- a/docs/source/api/catalog.rst +++ b/docs/source/api/catalog.rst @@ -30,3 +30,12 @@ Interface .. doxygenclass:: paimon::Identifier :members: :undoc-members: + +.. doxygenclass:: paimon::Database + :members: + +.. doxygenclass:: paimon::Table + :members: + +.. doxygenstruct:: paimon::CatalogOptions + :members: diff --git a/docs/source/build_system.rst b/docs/source/build_system.rst index 6687290ce..44495f73c 100644 --- a/docs/source/build_system.rst +++ b/docs/source/build_system.rst @@ -102,6 +102,8 @@ Paimon provides a set of built-in optional plugins that you can link to as neede - ``Paimon::paimon_avro_file_format_shared`` - ``Paimon::paimon_blob_file_format_shared`` - ``Paimon::paimon_lance_file_format_shared`` + - ``Paimon::paimon_mosaic_file_format_shared`` + - ``Paimon::paimon_vortex_file_format_shared`` - File system plugins: @@ -113,8 +115,17 @@ Paimon provides a set of built-in optional plugins that you can link to as neede - Index plugins: - ``Paimon::paimon_file_index_shared`` + - ``Paimon::paimon_global_index_shared`` - ``Paimon::paimon_lumina_index_shared`` - ``Paimon::paimon_lucene_index_shared`` + - ``Paimon::paimon_full_text_index_shared`` + +.. note:: + + A plugin target only exists when its component is enabled. ``parquet``, + ``blob``, ``local``, ``file_index`` and ``global_index`` are always built; the + others require the matching ``PAIMON_ENABLE_*`` flag described in + :ref:`cpp_build_optional_components`. .. note:: diff --git a/docs/source/building.rst b/docs/source/building.rst index 93a4890b3..b0fe95e44 100644 --- a/docs/source/building.rst +++ b/docs/source/building.rst @@ -173,12 +173,20 @@ Optional Components By default, the C++ build system creates a fairly minimal build. We have several optional system components which you can opt into building by passing -boolean flags to ``cmake``. +boolean flags to ``cmake``. ORC and Avro are the exceptions: both already default +to ``ON``, while every other component below defaults to ``OFF``. * ``-DPAIMON_ENABLE_ORC=ON``: Paimon integration with Apache ORC * ``-DPAIMON_ENABLE_AVRO=ON``: Apache Avro libraries and Paimon integration * ``-DPAIMON_ENABLE_LANCE=ON``: Lance 0.39 file format support through a Rust FFI; requires Rust 1.88 or newer, cbindgen, and protoc 3.12 or newer +* ``-DPAIMON_ENABLE_MOSAIC=ON``: Apache Paimon Mosaic file format support through a Rust FFI; + requires cargo on ``PATH`` +* ``-DPAIMON_ENABLE_VORTEX=ON``: Vortex file format support through a Rust FFI; requires cargo, + the stable Rust toolchain (Vortex pins ``rust-version`` 1.95 or newer and is built with + ``RUSTUP_TOOLCHAIN=stable``), and a ``flatc`` whose version matches the flatbuffers crate that + Vortex uses. Distribution ``flatc`` packages are typically too old; set ``FLATC`` to the path of + a suitable binary when it is not on ``PATH``. * ``-DPAIMON_ENABLE_JINDO=ON``: Support for Alibaba Jindo filesystems * ``-DPAIMON_ENABLE_OSS=ON``: Support for Alibaba Cloud OSS through OSS SDK V2 * ``-DPAIMON_ENABLE_S3=ON``: Support for Amazon S3-compatible filesystems @@ -278,6 +286,8 @@ For development builds, you will often want to enable additional targets in enable to exercise your changes, using the following ``cmake`` options. * ``-DPAIMON_BUILD_TESTS=ON``: Build executable unit tests. +* ``-DPAIMON_ENABLE_NETWORK_TESTS=ON``: Enable tests that access real remote services over the + network. These are disabled by default. Optional Checks ~~~~~~~~~~~~~~~ diff --git a/docs/source/user_guide/catalog.rst b/docs/source/user_guide/catalog.rst index 2d789f839..88e1eecd8 100644 --- a/docs/source/user_guide/catalog.rst +++ b/docs/source/user_guide/catalog.rst @@ -158,6 +158,13 @@ offsets when a real-time writer is created: PAIMON_ASSIGN_OR_RAISE(std::unique_ptr writer, paimon::FileStoreWrite::Create(std::move(write_context))); +The ``Table`` handle is an equivalent entry point: ``table->NewCommit()``, +``table->NewWrite()``, ``table->NewRead()`` and ``table->NewScan()`` each take the +same finished context and return the same operator as ``FileStoreCommit::Create()``, +``FileStoreWrite::Create()``, ``TableRead::Create()`` and ``TableScan::Create()``. +Pick whichever reads better; the context still carries the catalog when one is +needed. See the :ref:`Table API `. + For real-time writes, also set ``WithStreamingMode(true)`` and ``WithRealtimeContext(...)`` with ``realtime.enabled=true``. After ``CommitWithProgress`` succeeds, pass its returned snapshot id to @@ -431,12 +438,22 @@ file system of your own when the credentials expire. class MyCredentialProvider : public paimon::CredentialProvider { public: + // The factory hands over the path and the file system options of the accesses to + // authenticate, which is where you read your own configuration from. + MyCredentialProvider(std::string path, std::map options) + : path_(std::move(path)), options_(std::move(options)) {} + // Called as each access is signed; reload the credentials before they expire. paimon::Result> GetCredentials() const override { + // Request fresh credentials for path_ from the token service in options_. return std::map{ {"fs.oss.accessKeyId", ...}, {"fs.oss.accessKeySecret", ...}, {"fs.oss.securityToken", ...}}; } + + private: + std::string path_; + std::map options_; }; class MyCredentialProviderFactory : public paimon::CredentialProviderFactory { diff --git a/docs/source/user_guide/compaction.rst b/docs/source/user_guide/compaction.rst index 782fd53fd..b80bfdee4 100644 --- a/docs/source/user_guide/compaction.rst +++ b/docs/source/user_guide/compaction.rst @@ -78,8 +78,8 @@ Full Compaction Full compaction rewrites all eligible files in the bucket. During full compaction: -- Files whose size is already at or above ``compaction.file-size`` (and have no - associated deletion vectors) are skipped to avoid unnecessary rewrites. +- Files whose size is already at or above the compaction file-size threshold (and + have no associated deletion vectors) are skipped to avoid unnecessary rewrites. - When deletion vectors are enabled, all files are always eligible for compaction regardless of size, because deletion vectors must be applied. - When ``compaction.force-rewrite-all-files`` is ``true``, all files are @@ -88,10 +88,17 @@ compaction: small files exceeds the number of large files and the total file count is at least 3. -After compaction, if the last output file is still smaller than -``compaction.file-size``, it is placed back into the compaction queue for future +After compaction, if the last output file is still smaller than the compaction +file-size threshold, it is placed back into the compaction queue for future merging. +The compaction file-size threshold is not a separate option. It is derived from +``target-file-size`` as ``target-file-size / 10 * 7``, that is 70% of the target +size, so that a file already close to the target is not compacted twice. +``target-file-size`` defaults to 256 MiB for append-only tables and 128 MiB for +primary-key tables, which makes the derived threshold about 179 MiB and 89 MiB +respectively. Adjust ``target-file-size`` to move the threshold. + Dictionary Passthrough ~~~~~~~~~~~~~~~~~~~~~~ An append-only compaction rewrite copies rows into the new file without @@ -126,7 +133,7 @@ dictionary-encodes it the same way; this option only selects ``STRING`` columns. A file with a serialized Arrow dictionary schema can independently produce a ``BINARY`` dictionary, which the value accessors also support. The rewrite also overrides the option back to ``false`` when the table writes a format other than Parquet, when -``parquet.enable-dictionary`` is ``false`` because the writer would only expand +``parquet.enable.dictionary`` is ``false`` because the writer would only expand the values again, or when variant/map shredding is configured because those writers reshape each batch against a fixed physical schema. Setting the option therefore never makes a rewrite fail; at worst it has no effect. diff --git a/docs/source/user_guide/manifest_entry_cache.rst b/docs/source/user_guide/manifest_entry_cache.rst index 995d77e38..4d9fed056 100644 --- a/docs/source/user_guide/manifest_entry_cache.rst +++ b/docs/source/user_guide/manifest_entry_cache.rst @@ -86,6 +86,14 @@ under .AddOption(Options::SCAN_MANIFEST_ENTRY_CACHE_MAX_SNAPSHOTS, "3") .Finish()); +.. note:: + + ``LruCache`` is the built-in weight-bounded implementation, but it is declared in + ``src/paimon/common/io/cache/lru_cache.h`` and only ``include/`` is installed, so it is + not reachable from an installed Paimon. ``WithCache`` takes the public ``Cache`` + interface of ``include/paimon/cache/cache.h``; pass your own implementation of it when + you build against the installed package. + Cache entries are scoped by table path, branch, and bucket, so they can be reused across newly created ``TableScan`` and ``FileStoreScan`` instances as long as they share the same cache object and scan the same bucket. diff --git a/docs/source/user_guide/primary_key_global_index.rst b/docs/source/user_guide/primary_key_global_index.rst index 9307d938d..e955cd43f 100644 --- a/docs/source/user_guide/primary_key_global_index.rst +++ b/docs/source/user_guide/primary_key_global_index.rst @@ -84,10 +84,11 @@ Current scope definitions are recognized for validation, but their evaluation conservatively falls back to a normal scan until their dedicated payload readers are supported. - The read path targets the Java release-2.0.0 layout and scan semantics (source metadata - v1, ``GlobalIndexMeta`` with ``_SOURCE_META``, commit message v12). Source-file names - currently use the existing C++ length-prefixed UTF-8 streams; ASCII and non-null BMP - names are compatible with Java ``writeUTF``, while complete modified UTF-8 support for - supplementary code points will be handled by a shared stream-level change. + v1, ``GlobalIndexMeta`` with ``_SOURCE_META``). Commit messages are written at version 13 + and remain readable down to version 3. Source-file names currently use the existing C++ + length-prefixed UTF-8 streams; ASCII and non-null BMP names are compatible with Java + ``writeUTF``, while complete modified UTF-8 support for supplementary code points will be + handled by a shared stream-level change. - Automatic maintenance is synchronous during prepare-commit. Java's asynchronous build scheduling, retries, fairness metrics, and manual rebuild actions are not part of the C++ API. Realtime and postpone-bucket writers do not build source-backed payloads. diff --git a/docs/source/user_guide/system_tables.rst b/docs/source/user_guide/system_tables.rst index 6ac018823..843640eef 100644 --- a/docs/source/user_guide/system_tables.rst +++ b/docs/source/user_guide/system_tables.rst @@ -26,6 +26,59 @@ Branch-qualified paths are also supported. For example, ``/warehouse/db.db/orders$branch_audit$files`` reads the files system table from the ``audit`` branch. +Table-Level System Tables +------------------------- + +These suffixes are recognized on a data table path. Matching is +case-insensitive, and an unrecognized suffix fails with +``unsupported system table: `` instead of falling back to the base table. + +* ``$options``: the latest base table options as key/value rows. +* ``$snapshots``: snapshot commit history. +* ``$schemas``: schema evolution history. +* ``$tags``: tags and the snapshots they reference. +* ``$branches``: table branches, including ``main``. +* ``$consumers``: persisted streaming consumer offsets. +* ``$manifests``: data manifest metadata in the latest snapshot. +* ``$files``: data file metadata in the latest snapshot. +* ``$audit_log``: row-level changelog records with rowkind. +* ``$binlog``: changelog records with list-wrapped data columns. +* ``$ro``: read-optimized data, described in the next section. + +Global System Tables +-------------------- + +Four system tables are not bound to a single data table. They live under the +reserved ``sys`` database, and three of them enumerate across all databases in +the warehouse: + +* ``sys.catalog_options``: catalog-level configuration as key/value rows. +* ``sys.all_table_options``: all table options as + ``(database_name, table_name, key, value)`` rows. +* ``sys.tables``: metadata for all tables, including record counts and file + statistics. +* ``sys.partitions``: partition-level file statistics for all tables. + +``sys.all_table_options``, ``sys.tables`` and ``sys.partitions`` are always +available. ``sys.catalog_options`` is disabled by default and has to be turned +on with the catalog option ``catalog-options-table.enabled``; a disabled one is +reported as not existing. + +Those three enumerate databases and tables, so they need a catalog: loading a +bare ``/sys/`` path fails with ``global system table requires +catalog context: ``. ``sys.catalog_options`` enumerates nothing, and is +the one a bare path serves. + +.. note:: + + No public read path serves those three yet. A catalog lists them and + returns their schema, but does not hand back a readable table: + ``Catalog::GetTable(Identifier("sys", "tables"))`` is rejected by a + file-system catalog with ``do not support checking TableSchemaExists for + system table``, and a REST catalog returns one whose path is left unset, + which ``TableScan`` and ``TableRead`` cannot dispatch to the global system + table loader. + Read-Optimized System Table --------------------------- diff --git a/docs/source/user_guide/write.rst b/docs/source/user_guide/write.rst index 4a3d27dbe..127889ad1 100644 --- a/docs/source/user_guide/write.rst +++ b/docs/source/user_guide/write.rst @@ -231,9 +231,15 @@ produce a correct ``Snapshot``, which commonly includes (but is not limited to): The C++ writer supports Append and PK tables and can produce ``CommitMessage`` objects for both. ``FileStoreCommit`` supports direct file-system commits for both table types on non-object-store paths. - Object-store paths require REST catalog commit mode. Changelog is out of - scope and should not be emitted in ``CommitMessage`` until explicitly - supported. + Object-store paths need a commit that does not rename: either a catalog that + manages the versions of its tables, such as the REST catalog, passed to + ``CommitContextBuilder::WithCatalog()``, or ``UseRESTCatalogCommit(true)``, + which builds the commit request for the caller to send instead of publishing + the snapshot itself. The two are mutually exclusive. A catalog without + version management still commits by renaming, so passing one is rejected on + an object-store path. Changelog files are produced on the write path and + committed with their snapshot, except by the overwrite flow, which ignores + them and logs a warning naming the files it ignored. Serialization and Deserialization ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -256,7 +262,7 @@ Operational Flow 1. Writer nodes perform data ingestion and produce Arrow ``RecordBatch`` organized by partition and bucket. -2. Writers flush batches into ORC, Parquet, or Avro files via registered +2. Writers flush batches into data files through the registered ``file.format`` and ``file-system`` backends, producing file-level metadata and per-batch commit state. diff --git a/include/paimon/data/shredding/map_shared_shredding_schema_utils.h b/include/paimon/data/shredding/map_shared_shredding_schema_utils.h index bce0b6534..50eb123e1 100644 --- a/include/paimon/data/shredding/map_shared_shredding_schema_utils.h +++ b/include/paimon/data/shredding/map_shared_shredding_schema_utils.h @@ -64,12 +64,14 @@ struct PAIMON_EXPORT MapSharedShreddingFieldMeta { /// /// Example: read keys "age" and "score" from MAP column `attributes`: /// -/// auto builder = MapSharedShreddingAccessBuilder::Create(attributes_field); -/// builder->AddKey("age"); -/// builder->AddKey("score"); -/// auto field = builder->Build(); +/// PAIMON_ASSIGN_OR_RAISE(std::unique_ptr builder, +/// MapSharedShreddingAccessBuilder::Create(attributes_field)); +/// PAIMON_RETURN_NOT_OK(builder->AddKey("age")); +/// PAIMON_RETURN_NOT_OK(builder->AddKey("score")); +/// PAIMON_ASSIGN_OR_RAISE(std::unique_ptr field, builder->Build()); /// -/// Use the returned field in `ReadContextBuilder::SetReadSchema`. +/// Use the resulting field in `ReadContextBuilder::SetReadSchema`. Both macros require the +/// enclosing function to return `Status` or `Result`. class PAIMON_EXPORT MapSharedShreddingAccessBuilder { public: /// Creates a builder bound to the original MAP field. diff --git a/include/paimon/data/variant.h b/include/paimon/data/variant.h index 1958d09de..d93195856 100644 --- a/include/paimon/data/variant.h +++ b/include/paimon/data/variant.h @@ -167,11 +167,13 @@ class PAIMON_EXPORT Variant { /// Example: read `$.age` as INT64 and `$.city` as STRING from variant column `v`: /// /// VariantAccessBuilder builder; -/// builder.AddField(age_type, "$.age"); -/// builder.AddField(city_type, "$.city"); -/// auto field = builder.Build("v"); // use in ReadContextBuilder::SetReadSchema +/// PAIMON_RETURN_NOT_OK(builder.AddField(age_type, "$.age")); +/// PAIMON_RETURN_NOT_OK(builder.AddField(city_type, "$.city")); +/// PAIMON_ASSIGN_OR_RAISE(std::unique_ptr field, builder.Build("v")); /// -/// The resulting struct has one child per added field, named by its position ("0", "1", ...). +/// Use the resulting field in `ReadContextBuilder::SetReadSchema`. Both macros require the +/// enclosing function to return `Status` or `Result`. The resulting struct has one child +/// per added field, named by its position ("0", "1", ...). class PAIMON_EXPORT VariantAccessBuilder { public: VariantAccessBuilder(); diff --git a/include/paimon/read_context.h b/include/paimon/read_context.h index e62386c65..81b0f8fa2 100644 --- a/include/paimon/read_context.h +++ b/include/paimon/read_context.h @@ -311,7 +311,10 @@ class PAIMON_EXPORT ReadContextBuilder { /// }); /// /// auto c_schema = std::make_unique(); - /// arrow::ExportSchema(*projected_schema, c_schema.get()); + /// arrow::Status export_status = arrow::ExportSchema(*projected_schema, c_schema.get()); + /// if (!export_status.ok()) { + /// return paimon::Status::Invalid(export_status.message()); + /// } /// /// ReadContextBuilder builder("/path/to/table"); /// builder.SetReadSchema(std::move(c_schema)); diff --git a/src/paimon/common/data/blob_defs.h b/src/paimon/common/data/blob_defs.h index ce5932b43..8dbb681b1 100644 --- a/src/paimon/common/data/blob_defs.h +++ b/src/paimon/common/data/blob_defs.h @@ -40,7 +40,7 @@ class BlobDefs { /// To create a Blob field: /// @code /// std::unordered_map blob_metadata_map = { - /// {Blob::kExtensionTypeKey, Blob::kExtensionTypeValue} + /// {BlobDefs::kExtensionTypeKey, BlobDefs::kExtensionTypeValue} /// }; /// auto field = arrow::field("my_blob_field", arrow::large_binary(), false, /// std::make_shared(blob_metadata_map)); From f44c42989cf09541a3cb6b8f8dfd5acf84610225 Mon Sep 17 00:00:00 2001 From: "yonghao.fyh" Date: Mon, 2 Nov 2026 11:35:43 +0800 Subject: [PATCH 2/3] fix --- README.md | 2 +- docs/source/user_guide/compaction.rst | 5 ++++- docs/source/user_guide/primary_key_table.rst | 4 +++- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index e6d4e5a5e..d59dfb3fa 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ Paimon C++ currently provides: - **Write**: append table and primary key table write support with compaction. - **Commit**: append table and primary key table commit support. Direct file-system commits work on non-object-store paths; object-store paths require a catalog-mediated commit. -- **Scan**: batch and stream scan for append tables and primary key tables without changelog. +- **Scan**: batch and stream scan for append tables and primary key tables, with changelog consumption when a changelog producer is configured. - **Read**: append table read, primary key table read with deletion vector, and primary key table merge-on-read. - **Arrow integration**: batch read and write interfaces based on the [Arrow Columnar In-Memory Format](https://arrow.apache.org). - **File systems**: file system abstraction with built-in local, Jindo, OSS, and S3 file system support. diff --git a/docs/source/user_guide/compaction.rst b/docs/source/user_guide/compaction.rst index b80bfdee4..92aa01c81 100644 --- a/docs/source/user_guide/compaction.rst +++ b/docs/source/user_guide/compaction.rst @@ -26,7 +26,10 @@ compaction is therefore a trade-off between write throughput and read efficiency .. note:: - There can only be one job working on the same partition's compaction, otherwise it will cause conflicts. - - Paimon C++ does not support producing changelog for now. + - Changelog is produced according to the ``changelog-producer`` table option + (``input``, ``full-compaction`` or ``lookup``): the ``full-compaction`` and + ``lookup`` producers emit their changelog files from compaction, and the + files are committed with their snapshot. - Automatic compaction is disabled when ``write-only`` is set to ``true``, or when an append-only table uses unaware-bucket mode (``bucket = -1``). - For a complete list of compaction-related configurations, see the diff --git a/docs/source/user_guide/primary_key_table.rst b/docs/source/user_guide/primary_key_table.rst index e9cdc10b3..ed132bc95 100644 --- a/docs/source/user_guide/primary_key_table.rst +++ b/docs/source/user_guide/primary_key_table.rst @@ -38,7 +38,9 @@ efficient querying. Each bucket directory contains an LSM tree and its changelog files. .. note:: - Changelog is not supported yet for Paimon C++ primary key table write. + Primary key table writes produce changelog according to the + ``changelog-producer`` table option, and the changelog files are committed + with their snapshot. The range for a bucket is determined by the hash value of one or more columns in the records. Users can specify bucketing columns by providing the bucket-key option. From 639101f85fef7d40c43135f6a60343b94ac95cad Mon Sep 17 00:00:00 2001 From: "yonghao.fyh" Date: Mon, 2 Nov 2026 12:00:26 +0800 Subject: [PATCH 3/3] docs: fix outdated changelog-unsupported note in write guide --- docs/source/user_guide/write.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/source/user_guide/write.rst b/docs/source/user_guide/write.rst index 127889ad1..9d14c2414 100644 --- a/docs/source/user_guide/write.rst +++ b/docs/source/user_guide/write.rst @@ -31,8 +31,12 @@ Parquet, and Avro, thereby improving write throughput. - Append table - Primary Key table - Not supported in the current scope: - - Changelog + Changelog is produced on the write path according to the + ``changelog-producer`` table option (``input``, ``full-compaction`` or + ``lookup``), and the changelog files are committed with their snapshot, + except by the overwrite flow, which ignores them and logs a warning naming + the files it ignored. See :doc:`compaction` for the ``full-compaction`` and + ``lookup`` producers. Bucketing Modes ---------------