From 1659ef670bdec4a78c6587fa9c0c935aa52e3573 Mon Sep 17 00:00:00 2001 From: gstamatakis95 <126914070+gstamatakis95@users.noreply.github.com> Date: Fri, 28 Aug 2026 11:31:11 +0300 Subject: [PATCH] [docs] Document Paimon tiering issue with externally evolved schema Document the known issue (#4038) where external schema changes on a Fluss-managed Paimon table stall the tiering service. Closes #4138. --- website/docs/engine-flink/ddl.md | 4 ++++ .../datalake-formats/paimon.md | 19 +++++++++++++++++++ .../streaming-lakehouse/tiering-service.md | 4 ++++ 3 files changed, 27 insertions(+) diff --git a/website/docs/engine-flink/ddl.md b/website/docs/engine-flink/ddl.md index 990cd00b31e..37ca0090a9a 100644 --- a/website/docs/engine-flink/ddl.md +++ b/website/docs/engine-flink/ddl.md @@ -255,6 +255,10 @@ ALTER TABLE MyTable ADD ( ); ``` +:::note +For tables with `'table.datalake.enabled' = 'true'` on a Paimon or Iceberg lakehouse, Fluss also applies the added columns to the corresponding lake table as part of the `ALTER TABLE` statement, keeping both schemas in sync. Adding columns to datalake-enabled Hudi and Lance tables is not yet supported. Never add columns to the lake table directly through an external engine, since a lake table schema that diverges from the Fluss table schema stops the tiering service. See [Schema Evolution](../streaming-lakehouse/datalake-formats/paimon.md#schema-evolution) for details. +::: + ### Add watermark Assuming that `my_table` already contains a timestamp column named `ts`, the following SQL adds a watermark with strategy `ts - INTERVAL '10' SECOND`. The `ts` column is then used as the event-time attribute of the table. diff --git a/website/docs/streaming-lakehouse/datalake-formats/paimon.md b/website/docs/streaming-lakehouse/datalake-formats/paimon.md index de56df35db5..0560499cc82 100644 --- a/website/docs/streaming-lakehouse/datalake-formats/paimon.md +++ b/website/docs/streaming-lakehouse/datalake-formats/paimon.md @@ -180,6 +180,25 @@ SELECT COUNT(*) FROM paimon_catalog.fluss.orders; SELECT * FROM paimon_catalog.fluss.enriched_orders$snapshots; ``` +## Schema Evolution + +The schema of a Paimon table managed by Fluss must always be evolved through Fluss. When you add columns to a Fluss table with `ALTER TABLE ... ADD` (see [Add Columns](../../engine-flink/ddl.md#add-columns)), the new columns are appended at the end as nullable columns, and Fluss applies the same change to the Paimon table as part of the `ALTER TABLE` statement, so the two schemas stay in sync. + +:::warning External schema changes stall tiering +Do not change the schema of a Fluss-managed Paimon table through an external engine, for example by adding a column on the Paimon table from Spark or Doris. The tiering service requires the user columns of the Paimon table to match the Fluss table schema. Once the Paimon table contains a column that the Fluss table does not have, tiering the table's records can fail with an error like the following: + +``` +Caused by: java.io.IOException: Failed to write Fluss record to Paimon. +Caused by: java.lang.IllegalStateException: Field 18 is NULL because Paimon schema is wider than Fluss record. +``` + +The tiering job then fails and restarts in a loop until the schemas match again, cycling between the RUNNING and RESTARTING states in the Flink UI. The restart loop also interrupts tiering progress for the other tables served by the same job. Adding a column is the most common trigger, but any external change that makes the schemas diverge stops tiering in the same way, possibly with a different error message. + +No data is lost or corrupted, and the records that could not be tiered remain readable in Fluss. However, Fluss keeps log data until it has been tiered, so log retention for the table is effectively paused and storage usage grows until the schemas match again. Fix the mismatch promptly. + +To recover, make the two schemas consistent again. To keep the externally added column, run a matching `ALTER TABLE ... ADD` statement on the Fluss table. When the Paimon table already contains the column in the expected position, Fluss completes the change without touching the Paimon table. If Fluss rejects the statement because the schemas cannot be reconciled, drop the externally added column from the Paimon table and, if you still need it, add it through Fluss afterwards. Once the schemas match, the tiering job recovers on its next automatic restart and the pending records are tiered completely. If you cancelled the tiering job in the meantime, resubmit it. +::: + ## Data Type Mapping When integrating with Paimon, Fluss automatically converts between Fluss data types and Paimon data types. diff --git a/website/docs/streaming-lakehouse/tiering-service.md b/website/docs/streaming-lakehouse/tiering-service.md index 93ad1914bba..2494902cd73 100644 --- a/website/docs/streaming-lakehouse/tiering-service.md +++ b/website/docs/streaming-lakehouse/tiering-service.md @@ -15,6 +15,10 @@ The Tiering Service is implemented as an Apache Flink job that: - Maintains exactly-once semantics between Fluss and the data lake - Operates incrementally, syncing only missing data segments +:::warning +Never change the schema of a lake table managed by the Tiering Service through an external engine. The Tiering Service requires the lake table schema to stay in sync with the Fluss table schema, and a diverging schema makes the tiering job fail and restart in a loop until the schemas match again. For lake formats that support schema evolution through Fluss (Paimon and Iceberg), add columns with `ALTER TABLE` on the Fluss table instead. See [Schema Evolution](datalake-formats/paimon.md#schema-evolution) for the symptoms and the recovery steps with Paimon. +::: + For an in-depth look at the Tiering Service internals, see the blog series: - [Tiering Service Deep Dive — Part 1](https://fluss.apache.org/blog/fluss-tiering-service-deep-dive-part1/)