Skip to content

Commit a66b40d

Browse files
committed
fix(scan): restore parameter order and document cache refresh
1 parent dccf25e commit a66b40d

4 files changed

Lines changed: 20 additions & 13 deletions

File tree

‎docs/source/api/scan.rst‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ are rejected. The physical table path must match, including when scanning its ``
7272
``$audit_log`` system table. Format tables and global system tables do not use these resources.
7373
``Finish()`` resets the builder's resource setting, like ``WithCache()`` and ``WithExecutor()``.
7474

75-
Every new scan still checks for the latest schema. Previously loaded schema versions, Arrow
75+
Every new scan still checks for the latest schema ID. Previously loaded schema versions, Arrow
7676
schemas, partition and primary-key field information, and statistics evolution objects are
7777
reused. Existing scans retain their original schema. Snapshot selection, filters, streaming
7878
progress, executors and scan metrics remain independent. ``SetTableSchema()`` keeps its existing
@@ -81,9 +81,14 @@ behavior: on main it bypasses the shared schema cache; on other branches it is i
8181
The resources can be shared by concurrent scans when the supplied file system and metadata
8282
memory pool support concurrent use. The resource object retains its metadata pool independently
8383
of each scan's memory pool. Cached schema versions are retained for the resource object's lifetime;
84-
there is no automatic eviction or background refresh. Recreate the resources after deleting and
85-
recreating a table or branch, or changing file system access configuration. A metadata cache does
86-
not pin snapshots or prevent their data files from expiring.
84+
there is no automatic eviction or background refresh. The caller must recreate the resources after
85+
fast-forward, deleting and recreating a table or branch, or changing file system access configuration.
86+
Fast-forward can replace schema contents under existing IDs, so discovering the latest ID does not
87+
refresh a cached schema. After fast-forward completes, create a new ``TableScanResources`` and use it
88+
for subsequent scans; existing resources and scans retain their cached metadata. This follows
89+
Paimon's `fast-forward cache refresh requirement
90+
<https://paimon.apache.org/docs/1.3/maintenance/manage-branches/#fast-forward>`_.
91+
A metadata cache does not pin snapshots or prevent their data files from expiring.
8792

8893
Interface
8994
=========

‎include/paimon/scan_context.h‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,14 +51,14 @@ class PAIMON_EXPORT ScanContext {
5151
const std::shared_ptr<ScanFilter>& scan_filter,
5252
const std::shared_ptr<GlobalIndexResult>& global_index_result,
5353
const std::shared_ptr<RealtimeContext>& realtime_context,
54+
const std::shared_ptr<MemoryPool>& memory_pool,
5455
const std::shared_ptr<Executor>& executor,
5556
const std::shared_ptr<FileSystem>& specific_file_system,
5657
const std::optional<std::string>& table_schema,
5758
const std::map<std::string, std::string>& options,
5859
const std::shared_ptr<Cache>& cache,
5960
const std::shared_ptr<FormatTable>& format_table,
60-
const std::shared_ptr<TableScanResources>& table_resources,
61-
const std::shared_ptr<MemoryPool>& memory_pool);
61+
const std::shared_ptr<TableScanResources>& table_resources);
6262

6363
~ScanContext();
6464

‎include/paimon/table/source/table_scan_resources.h‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -33,9 +33,11 @@ class TableScanResourcesAccess;
3333
/// Metadata resources shared by scans of one managed table and branch.
3434
///
3535
/// Share this object through ScanContextBuilder::WithTableResources(). Each scan still discovers
36-
/// the latest schema and chooses its own snapshot. Schema versions and their derived metadata
37-
/// remain cached for the lifetime of this object. Recreate it after dropping and recreating a
38-
/// table or branch, or changing the file system's access configuration.
36+
/// the latest schema ID and chooses its own snapshot. Schema versions and their derived metadata
37+
/// remain cached for the lifetime of this object. The caller must recreate it after fast-forward,
38+
/// dropping and recreating a table or branch, or changing the file system's access configuration.
39+
/// Fast-forward can replace schema contents under existing IDs; cached versions are not refreshed
40+
/// automatically. Use the new resources for subsequent scans.
3941
///
4042
/// Concurrent scans may share these resources. The supplied file system and metadata memory pool
4143
/// must support concurrent use. Filters, executors, scan progress and scan metrics are not shared.

‎src/paimon/core/operation/scan_context.cpp‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -36,14 +36,14 @@ ScanContext::ScanContext(const std::string& path, bool is_streaming_mode,
3636
const std::shared_ptr<ScanFilter>& scan_filter,
3737
const std::shared_ptr<GlobalIndexResult>& global_index_result,
3838
const std::shared_ptr<RealtimeContext>& realtime_context,
39+
const std::shared_ptr<MemoryPool>& memory_pool,
3940
const std::shared_ptr<Executor>& executor,
4041
const std::shared_ptr<FileSystem>& specific_file_system,
4142
const std::optional<std::string>& table_schema,
4243
const std::map<std::string, std::string>& options,
4344
const std::shared_ptr<Cache>& cache,
4445
const std::shared_ptr<FormatTable>& format_table,
45-
const std::shared_ptr<TableScanResources>& table_resources,
46-
const std::shared_ptr<MemoryPool>& memory_pool)
46+
const std::shared_ptr<TableScanResources>& table_resources)
4747
: path_(path),
4848
is_streaming_mode_(is_streaming_mode),
4949
limit_(limit),
@@ -249,9 +249,9 @@ Result<std::unique_ptr<ScanContext>> ScanContextBuilder::Finish() {
249249
impl_->path_, impl_->is_streaming_mode_, impl_->limit_,
250250
std::make_shared<ScanFilter>(impl_->predicates_, impl_->partition_filters_,
251251
impl_->bucket_filter_),
252-
impl_->global_index_result_, impl_->realtime_context_, executor,
252+
impl_->global_index_result_, impl_->realtime_context_, impl_->memory_pool_, executor,
253253
impl_->specific_file_system_, impl_->table_schema_, options, impl_->cache_,
254-
impl_->format_table_, impl_->table_resources_, impl_->memory_pool_);
254+
impl_->format_table_, impl_->table_resources_);
255255
impl_->Reset();
256256
return ctx;
257257
}

0 commit comments

Comments
 (0)