Skip to content

Commit 709bcfc

Browse files
authored
Merge branch 'main' into codex/pr437-moved-function-oracle-20261019
2 parents b4a1dab + 50ef548 commit 709bcfc

185 files changed

Lines changed: 8724 additions & 2447 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎LICENSE‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,8 @@ This product includes code from Alibaba Havenask.
207207

208208
* include/paimon/factories/singleton.h
209209
* src/paimon/common/factories/singleton.cpp
210+
* src/paimon/common/utils/hash_map.h
211+
* src/paimon/common/utils/dense_hash_table.h
210212

211213
Copyright: 2014-present Alibaba Inc.
212214
Home page: https://havenask.net/

‎docs/source/user_guide/data_types.rst‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -197,13 +197,27 @@ and `Arrow DataTypes <https://arrow.apache.org/docs/format/Columnar.html#data-ty
197197
Paimon C++ supports VECTOR value columns in append-only and primary-key
198198
tables backed by Parquet data files. They use the standard Parquet LIST
199199
representation on disk and are restored as Arrow ``FixedSizeList``
200-
values on read. Row-tracking append-only tables with data evolution also
201-
support full-row and partial-column VECTOR writes, including reads across
200+
values on read. With Vortex enabled at build time, Vortex data files also
201+
support VECTOR value columns. Row-tracking append-only Parquet tables
202+
with data evolution support full-row and partial-column VECTOR writes, including reads across
202203
schema versions that add or drop VECTOR columns. Changing a VECTOR's
203204
dimension or element type is not supported. VECTOR columns cannot be
204205
primary, partition, or bucket keys, nor comparator-based
205-
ordering fields such as sequence and sequence-group fields. Dedicated
206-
vector storage is not included yet.
206+
ordering fields such as sequence and sequence-group fields.
207+
208+
``vector.file.format`` stores a single top-level VECTOR column in dedicated
209+
``.vector.<format>`` files. Supported formats are ``parquet``, ``lance``,
210+
and ``vortex``; Lance and Vortex must be enabled at build time.
211+
This mode requires ``file.format=parquet``, ``bucket=-1``,
212+
``row-tracking.enabled=true``, and ``data-evolution.enabled=true``, plus at
213+
least one ordinary non-partition column. It supports full-row inserts into append-only
214+
tables. Custom write schemas, partial writes, multiple or nested VECTOR
215+
columns, and BLOB columns are not supported in this mode.
216+
217+
VECTOR files roll independently at ``vector.target-file-size``, which
218+
defaults to ``target-file-size``. Main files roll at ``target-file-size``
219+
or ``target-file-row-num`` and also close the current VECTOR file. One
220+
main file can therefore cover the combined row range of several VECTOR files.
207221

208222
Paimon C++ also reads Parquet files written by Paimon Rust or Python whose
209223
embedded Arrow schema restores VECTOR columns as ``FixedSizeList``,

‎docs/source/user_guide/global_index.rst‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,12 @@ BTree is an efficient index based on multi-level SST files, supporting rich pred
6060

6161
**Special Configuration:**
6262

63+
- **Option**: ``btree-index.bloom-filter.enabled``
64+
65+
- **Default**: ``false``
66+
- **Description**: Enables a Bloom filter for BTree point lookups such as equality and ``IN``
67+
predicates. This can avoid reading data blocks for missing keys.
68+
6369
- **Option**: ``btree-index.read-buffer-size``
6470

6571
- **Description**: Optional. Specifies the read buffer size for the B-tree index. This setting can be tuned based on query patterns:
@@ -168,9 +174,12 @@ index shard, the engine ranks matching rows by BM25 score and returns at most ``
168174
the highest scores. ``include_row_ids`` holds global row ids and restricts the rows that are
169175
ranked; an empty set matches no rows.
170176

171-
For both ``full-text`` and ``lucene-fts``, searches across multiple shards currently return the
172-
union of the shard candidates, which can exceed ``limit``. Applying a final global top-k is tracked
173-
in `issue #402 <https://github.com/apache/paimon-cpp/issues/402>`_.
177+
For both ``full-text`` and ``lucene-fts``, a search across multiple index shards keeps the
178+
``limit`` highest-scoring rows returned by the shards, breaking ties by ascending row id. Which rows
179+
tied at a shard's cutoff score the shard returns is up to its engine. A row in overlapping index
180+
ranges is ranked only by the range that starts first, or ends first if both start at the same row.
181+
A shard ignores the ``include_row_ids`` outside the rows it ranks and is not searched if none
182+
remain. The result iterates rows by ascending row id, not by score.
174183

175184
**Configuration:**
176185

‎examples/read_write_demo.cpp‎

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
#include "arrow/ipc/api.h"
2727
#include "paimon/api.h"
2828
#include "paimon/catalog/catalog.h"
29+
#include "paimon/catalog/table.h"
2930

3031
arrow::Result<std::shared_ptr<arrow::StructArray>> PrepareData(const arrow::FieldVector& fields) {
3132
arrow::StringBuilder f0_builder;
@@ -81,6 +82,11 @@ paimon::Status Run(const std::string& root_path, const std::string& db_name,
8182
/*primary_keys=*/{}, options,
8283
/*ignore_if_exists=*/false));
8384

85+
// The table every operator below is created from. Each call hands the table's shared owner
86+
// to the operator it creates.
87+
PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<paimon::Table> table,
88+
catalog->GetTable(paimon::Identifier(db_name, table_name)));
89+
8490
std::string table_path = root_path + "/" + db_name + ".db/" + table_name;
8591

8692
std::string commit_user = "some_commit_user";
@@ -89,7 +95,7 @@ paimon::Status Run(const std::string& root_path, const std::string& db_name,
8995
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::WriteContext> write_context,
9096
context_builder.SetOptions(options).Finish());
9197
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::FileStoreWrite> writer,
92-
paimon::FileStoreWrite::Create(std::move(write_context)));
98+
table->NewWrite(std::move(write_context)));
9399

94100
// prepare data
95101
auto struct_array = PrepareData(fields);
@@ -113,23 +119,23 @@ paimon::Status Run(const std::string& root_path, const std::string& db_name,
113119
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::CommitContext> commit_context,
114120
commit_context_builder.SetOptions(options).Finish());
115121
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::FileStoreCommit> committer,
116-
paimon::FileStoreCommit::Create(std::move(commit_context)));
122+
table->NewCommit(std::move(commit_context)));
117123
PAIMON_RETURN_NOT_OK(committer->Commit(commit_message));
118124

119125
// scan
120126
paimon::ScanContextBuilder scan_context_builder(table_path);
121127
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::ScanContext> scan_context,
122128
scan_context_builder.SetOptions(options).Finish());
123129
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::TableScan> scanner,
124-
paimon::TableScan::Create(std::move(scan_context)));
130+
table->NewScan(std::move(scan_context)));
125131
PAIMON_ASSIGN_OR_RAISE(std::shared_ptr<paimon::Plan> plan, scanner->CreatePlan());
126132

127133
// read
128134
paimon::ReadContextBuilder read_context_builder(table_path);
129135
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::ReadContext> read_context,
130136
read_context_builder.SetOptions(options).Finish());
131137
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::TableRead> table_read,
132-
paimon::TableRead::Create(std::move(read_context)));
138+
table->NewRead(std::move(read_context)));
133139
PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<paimon::BatchReader> batch_reader,
134140
table_read->CreateReader(plan->Splits()));
135141
arrow::ArrayVector result_array_vector;

‎include/paimon/catalog/identifier.h‎

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -27,16 +27,16 @@
2727

2828
namespace paimon {
2929

30-
/// An identifier for a table containing database and table name.
30+
/// An identifier for a catalog object: a database name plus an object name, which may carry a
31+
/// branch or system-table suffix.
3132
class PAIMON_EXPORT Identifier {
3233
public:
3334
static const char kUnknownDatabase[];
3435
static const char kSystemTableSplitter[];
3536
static const char kSystemBranchPrefix[];
3637
static const char kDefaultMainBranch[];
3738

38-
explicit Identifier(const std::string& table);
39-
Identifier(const std::string& database, const std::string& table);
39+
Identifier(const std::string& database, const std::string& object);
4040
/// Names `branch` of `table`: every branch but the main one is addressed by the object name
4141
/// this builds, which joins them as `tbl$branch_dev`.
4242
///
@@ -53,9 +53,15 @@ class PAIMON_EXPORT Identifier {
5353

5454
bool operator==(const Identifier& other) const;
5555
const std::string& GetDatabaseName() const;
56-
const std::string& GetTableName() const;
56+
/// The full object name, which keeps any branch and system-table suffix, like
57+
/// `tbl$branch_dev`. Mirrors the Java `Identifier#getObjectName`.
58+
const std::string& GetObjectName() const;
5759
Result<std::string> GetDataTableName() const;
5860
Result<std::optional<std::string>> GetBranchName() const;
61+
/// The branch the object name carries, `main` when it names none. Any spelling of the
62+
/// main branch, as `MAIN`, folds to `main` here, as the branch constructor folds it,
63+
/// so a caller binding a manager to the returned value addresses the main branch's own
64+
/// directory rather than one named after the alias.
5965
Result<std::string> GetBranchNameOrDefault() const;
6066
Result<std::optional<std::string>> GetSystemTableName() const;
6167
Result<bool> IsSystemTable() const;
@@ -67,10 +73,10 @@ class PAIMON_EXPORT Identifier {
6773
static Result<Identifier> FromString(const std::string& full_name);
6874

6975
private:
70-
Status SplitTableName() const;
76+
Status SplitObjectName() const;
7177

7278
const std::string database_;
73-
const std::string table_;
79+
const std::string object_;
7480
mutable bool parsed_ = false;
7581
mutable std::string data_table_;
7682
mutable std::optional<std::string> branch_;

‎include/paimon/catalog/table.h‎

Lines changed: 61 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -33,33 +33,50 @@
3333

3434
namespace paimon {
3535

36+
class TableScan;
37+
class TableRead;
38+
3639
/// A table provides basic abstraction for table type.
3740
class PAIMON_EXPORT Table {
3841
public:
39-
/// Creates a table by loading the latest schema from the specified table path.
40-
///
41-
/// @param file_system File system used to access the table directory and schema files.
42-
/// @param table_path Root path of the table in the file system.
43-
/// @param identifier Logical table identifier selecting the schema branch and table name.
44-
/// @return A table initialized with the latest available schema.
45-
static Result<std::shared_ptr<Table>> Create(const std::shared_ptr<FileSystem>& file_system,
46-
const std::string& table_path,
47-
const Identifier& identifier);
48-
42+
/// @param file_system File system the table's data is accessed through.
43+
/// @param root_path Root path of the table in the file system.
4944
/// @param uuid Metastore UUID, or empty if unavailable.
50-
Table(const std::shared_ptr<Schema>& schema, const std::string& database,
51-
const std::string& table_name, const std::string& uuid = "")
52-
: schema_(schema), database_(database), table_name_(table_name), uuid_(uuid) {}
45+
Table(const std::shared_ptr<FileSystem>& file_system, const std::string& root_path,
46+
const std::shared_ptr<Schema>& schema, const Identifier& identifier,
47+
const std::string& uuid = "");
5348

5449
~Table() = default;
5550

51+
/// Returns the file system the table was opened with, which serves the table's
52+
/// files. A catalog that hands out per-table file systems opens a table with the
53+
/// one for that table; it is null when the opener had none.
54+
std::shared_ptr<FileSystem> GetFileSystem() const {
55+
return file_system_;
56+
}
57+
58+
/// Returns the root path of the table in its file system. It is empty when the
59+
/// opener had no location for the table, which a mock in a test is one case of.
60+
const std::string& GetRootPath() const {
61+
return root_path_;
62+
}
63+
5664
/// A name to identify this table.
5765
std::string Name() const {
58-
return table_name_;
66+
return identifier_.GetObjectName();
5967
}
6068

6169
/// Full name of the table, default is database.tableName.
62-
std::string FullName() const;
70+
std::string FullName() const {
71+
return identifier_.GetFullName();
72+
}
73+
74+
/// Returns the branch the table was opened for, which its identifier carries; `main`
75+
/// when it names none or a spelling of it, which its managers are bound to. A
76+
/// malformed object name that hides the branch is reported.
77+
Result<std::string> CurrentBranch() const {
78+
return identifier_.GetBranchNameOrDefault();
79+
}
6380

6481
/// Returns the metastore UUID, falling back to `FullName()`.
6582
/// @note Use `CatalogUuid()` for commit requests to avoid sending a table name as an ID.
@@ -78,10 +95,37 @@ class PAIMON_EXPORT Table {
7895
return schema_;
7996
}
8097

98+
/// Creates a scan of this table from a finished scan context, the table-first entry point
99+
/// that `TableScan::Create()` also builds through. The scan is derived from the context.
100+
/// @param context The finished scan context.
101+
/// @return The scan, or the error building it reports.
102+
Result<std::unique_ptr<TableScan>> NewScan(std::unique_ptr<ScanContext> context) const;
103+
104+
/// Creates a read of this table from a finished read context, the table-first entry point
105+
/// that `TableRead::Create()` also builds through. The read is derived from the context.
106+
/// @param context The finished read context.
107+
/// @return The read, or the error building it reports.
108+
Result<std::unique_ptr<TableRead>> NewRead(std::unique_ptr<ReadContext> context) const;
109+
110+
/// Creates a write of this table from a finished write context, the table-first entry point
111+
/// that `FileStoreWrite::Create()` also builds through. The write is derived from the context.
112+
/// @param context The finished write context.
113+
/// @return The write, or the error building it reports.
114+
Result<std::unique_ptr<FileStoreWrite>> NewWrite(std::unique_ptr<WriteContext> context) const;
115+
116+
/// Creates a commit of this table from a finished commit context, the table-first entry point
117+
/// that `FileStoreCommit::Create()` also builds through. The commit is derived from the
118+
/// context.
119+
/// @param context The finished commit context.
120+
/// @return The commit, or the error building it reports.
121+
Result<std::unique_ptr<FileStoreCommit>> NewCommit(
122+
std::unique_ptr<CommitContext> context) const;
123+
81124
private:
125+
std::shared_ptr<FileSystem> file_system_;
126+
std::string root_path_;
82127
std::shared_ptr<Schema> schema_;
83-
std::string database_;
84-
std::string table_name_;
128+
Identifier identifier_;
85129
std::string uuid_;
86130
};
87131

‎include/paimon/defs.h‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -162,6 +162,14 @@ struct PAIMON_EXPORT Options {
162162
/// "blob.target-file-size" - Target size of a blob file. Default is TARGET_FILE_SIZE.
163163
static const char BLOB_TARGET_FILE_SIZE[];
164164

165+
/// "vector.file.format" - Optional dedicated VECTOR storage format (parquet, lance or vortex)
166+
/// for a single top-level VECTOR column in row-tracking append-only tables.
167+
static const char VECTOR_FILE_FORMAT[];
168+
169+
/// "vector.target-file-size" - Target size of the vector file. Defaults to TARGET_FILE_SIZE.
170+
/// VECTOR files roll independently; rolling a main file also closes the current VECTOR file.
171+
static const char VECTOR_TARGET_FILE_SIZE[];
172+
165173
/// "blob.split-by-file-size" - Whether to consider blob file size as a factor when performing
166174
/// scan splitting. When unset, defaults to the negation of BLOB_AS_DESCRIPTOR.
167175
static const char BLOB_SPLIT_BY_FILE_SIZE[];

‎include/paimon/global_index/global_index_io_meta.h‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,16 +24,22 @@
2424
#include <string>
2525

2626
#include "paimon/memory/bytes.h"
27+
#include "paimon/visibility.h"
2728

2829
namespace paimon {
2930
/// Metadata describing a single file entry in a global index.
3031
struct PAIMON_EXPORT GlobalIndexIOMeta {
31-
GlobalIndexIOMeta(const std::string& _file_path, int64_t _file_size,
32+
GlobalIndexIOMeta(const std::string& _file_path, int64_t _file_size, int64_t _row_count,
3233
const std::shared_ptr<Bytes>& _metadata)
33-
: file_path(_file_path), file_size(_file_size), metadata(_metadata) {}
34+
: file_path(_file_path),
35+
file_size(_file_size),
36+
row_count(_row_count),
37+
metadata(_metadata) {}
3438

3539
std::string file_path;
3640
int64_t file_size;
41+
/// Total logical rows processed by the writer, including null or skipped rows.
42+
int64_t row_count;
3743
/// Optional binary metadata associated with the file, such as serialized
3844
/// secondary index structures or inline index bytes.
3945
/// May be null if no additional metadata is available.

‎include/paimon/global_index/global_index_reader.h‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -45,9 +45,12 @@ class PAIMON_EXPORT GlobalIndexReader : public FunctionVisitor<std::shared_ptr<G
4545
virtual Result<std::shared_ptr<ScoredGlobalIndexResult>> VisitVectorSearch(
4646
const std::shared_ptr<VectorSearch>& vector_search) = 0;
4747

48-
/// VisitFullTextSearch performs full text search and returns row ids with relevance scores.
49-
/// Each index shard returns at most `full_text_search->limit` rows. Union readers currently
50-
/// retain all shard candidates and can return more than that limit.
48+
/// VisitFullTextSearch performs full text search and returns at most `full_text_search->limit`
49+
/// row ids with the highest relevance scores, iterated by ascending row id rather than score.
50+
/// A search over multiple index shards keeps the `limit` highest-scoring rows returned by the
51+
/// shards, breaking ties by ascending row id; which rows tied at a shard's cutoff score the
52+
/// shard returns is up to its index engine. A row in overlapping index ranges is ranked only by
53+
/// the range that starts first, or ends first if both start at the same row.
5154
/// @warning `VisitFullTextSearch` may return error status when it is incorrectly invoked (e.g.,
5255
/// BTreeGlobalIndexReader call `VisitFullTextSearch`) or the query is invalid.
5356
virtual Result<std::shared_ptr<ScoredGlobalIndexResult>> VisitFullTextSearch(

0 commit comments

Comments
 (0)