Skip to content

[lake/paimon] Introduce scan-based lake table lookuper - #4124

Open
zuston wants to merge 1 commit into
apache:mainfrom
zuston:scanBasedLookuper
Open

[lake/paimon] Introduce scan-based lake table lookuper#4124
zuston wants to merge 1 commit into
apache:mainfrom
zuston:scanBasedLookuper

Conversation

@zuston

@zuston zuston commented Aug 27, 2026

Copy link
Copy Markdown
Member

Purpose

this is the sub-task for #3631

The SST-based Paimon lake table lookuper introduced in #3632 caches Paimon SST files on TabletServer local disks. This approach can also benefit from future Fluss-level optimizations, such as a global index.

However, for highly sparse lookup workloads, the cache hit ratio can be very low, resulting in frequent file downloads and additional I/O overhead. Scan-based lookup can reduce this overhead by using predicate pushdown to read only the relevant Parquet row groups instead of entire files. This is particularly beneficial when row groups are small relative to the overall Parquet file size.

Brief change log

  1. introduce the scan-based lake table lookuper

Tests

API and Format

Documentation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant