[docs] Document clean lake schemas and rolling upgrade requirements - #4112
[docs] Document clean lake schemas and rolling upgrade requirements#4112fhan688 wants to merge 2 commits into
Conversation
luoyuxia
left a comment
luoyuxia
left a comment
There was a problem hiding this comment.
Static documentation review. I left four inline comments covering the remaining clean/legacy schema and upgrade-guidance inconsistencies.
| :::note | ||
| Iceberg tables created by earlier Fluss versions (**legacy** tables) still carry the three trailing system columns, together with the `__bucket` partitioning and `__offset` sort order described in the sections below. These tables are **not** migrated and remain fully readable and writable. Fluss detects the layout from the physical schema — a table is treated as legacy when it carries the system columns, and clean otherwise — so both layouts are supported side by side without any manual migration. | ||
|
|
||
| The `__bucket` partitioning, `__offset` sort order, and system columns shown in the **Primary Key Tables**, **Log Tables**, and **Partitioned Tables** sections below describe the **legacy** physical layout. A clean table has none of them. |
There was a problem hiding this comment.
Major: Could we update the mapping sections and examples below instead of only labeling them as legacy? For a table created by the shown DDL, the implementation derives the result step by step as follows: createIcebergSchema(..., false) emits only user columns; a clean table without a bucket key has no __bucket, so no identity(__bucket) partition is added; and createTable uses SortOrder.unsorted(). However, the Primary Key, Log, and Partitioned sections still call schemas containing the three system columns and SORTED BY (__offset) the "Corresponding Iceberg table". The general query example later also filters on __bucket/__offset, which do not exist on clean tables. Please make the clean layout the default result in these sections, move the old structures into an explicitly labeled legacy subsection if they are still useful, and replace or mark the system-column query as legacy-only.
| | Old tiering service | Supported | **Not supported** — must not process clean tables | | ||
| | Old Flink connectors using `FULL` startup mode | Readable | **Not readable** | | ||
|
|
||
| Old Flink connectors that use `FULL` startup mode assume the presence of the system columns and therefore cannot read newly created clean lake tables. |
There was a problem hiding this comment.
Major: Please call out that FULL is the default value of scan.startup.mode. FlinkConnectorOptions.SCAN_STARTUP_MODE uses defaultValue(ScanStartupMode.FULL), so an old connector with no explicit startup-mode setting is also incompatible with a clean lake table. Without that detail, operators may search only for jobs that explicitly set FULL and miss default-configured jobs before upgrading the cluster. It would also help to state that the connector and its matching lake storage plugin must be upgraded together; EARLIEST, LATEST, and TIMESTAMP avoid the lake bootstrap path, although switching modes changes startup semantics.
|
|
||
| The schema of the Iceberg table matches that of the Fluss table, except for the addition of three system columns at the end: `__bucket`, `__offset`, and `__timestamp`. | ||
| These system columns help Fluss clients consume data from Iceberg in a streaming fashion, such as seeking by a specific bucket using an offset or timestamp. | ||
| Newly created Iceberg tables (**clean** tables) contain only the user-defined columns of the Fluss table. Fluss no longer appends the `__bucket`, `__offset`, and `__timestamp` system columns to the physical schema, and a clean table therefore has no `__bucket` partitioning or `__offset` sort order. |
There was a problem hiding this comment.
Minor: Now that all newly created Iceberg tables are clean, the examples at lines 274 and 583 are misleading: both create a new table with table.datalake.auto-compaction = 'true', but IcebergLakeWriter schedules compaction only when autoCompaction && isLegacyTable; for a clean table it logs a warning and leaves the compaction executor null. A user copying either example may assume small-file maintenance is running when it is not. Please remove this option from clean-table CREATE TABLE examples, or demonstrate it only for an existing legacy table (for example via a clearly labeled ALTER TABLE example) and direct clean-table users to external Iceberg compaction.
|
|
||
| ## Lake Table Schema Changes (FIP-27) | ||
|
|
||
| Starting from this version, Fluss creates lake tables with a **clean** physical schema that contains only the user-defined columns. Earlier versions appended three trailing system columns (`__bucket`, `__offset`, `__timestamp`) to every lake table; these are no longer added to newly created tables. |
There was a problem hiding this comment.
Minor: The statement that earlier versions appended the three columns to "every lake table" is not true for Lance. Fluss v0.9 already shipped fluss-lake-lance; its LanceLakeCatalog creates fields directly from TableDescriptor.getSchema().getRowType() without appending system columns, and the v0.9 documentation says the Lance schema matches the Fluss schema. Please scope this schema change to Paimon and Iceberg, and mention that Lance already used the clean layout (while Hudi is clean-only because it was never publicly released).
3 participants
Purpose
Linked issue: close #3905
Sub-task of the FIP-27 umbrella (#2411): Remove Mandatory System Columns From Fluss Lake Tables.
The Paimon (#3982), Iceberg (#4019), and Hudi (#4077) implementation issues changed the physical schema of newly created lake tables to a clean layout containing only user-defined columns. The user-facing documentation, however, still described the old behavior (three system columns appended to every lake table) and did not cover the rolling-upgrade requirements this format change introduces.
This PR documents the user-visible schema behavior and the rolling-upgrade requirements of FIP-27, and brings the lake documentation in line with the delivered implementations.
Brief change log
datalake-formats/paimon.md: newly created Paimon tables are clean (user columns only); Paimon tables created by earlier versions (legacy) still carry the system columns, are not migrated, and remain supported. Fluss detects the layout from the physical schema. Fixes the outdated "adds three system columns" description.datalake-formats/iceberg.md: same clean/legacy schema description. Marks the__bucketidentity partitioning,__offsetsort order, and system columns in the Primary Key / Log / Partitioned table sections as the legacy physical layout; a clean table has none of them. Updates the "System Columns" section to state that only legacy tables carry them.datalake-formats/hudi.md: adds a note that the Hudi lake storage was never publicly released, so it only ever uses the clean layout and the legacy rolling-upgrade considerations do not apply to it.maintenance/operations/upgrade-notes-1.0.md: adds a "Lake Table Schema Changes (FIP-27)" section covering the clean and legacy layouts, a compatibility matrix, the required upgrade order (lake-reading Flink connectors and lake storage plugins → tiering service → Fluss cluster), theFULLstartup-mode read restriction, and rnotes that system columns remain disabled by default andthat any future opt-in is outside FIP-27.Tests
Documentation only; no tests. Verified that the internal resolve and that no outdated "appends system columns"descriptions remain in the docs.
API and Format
No code, API, or storage format change. This PR only documents the format behavior already delivered by #3982, #4019, and #4077.
Documentation
This PR is the documentation sub-task. It updates the Paie format pages and the v0.9 → v1.0 upgrade notes.