[docs] Document Paimon tiering issue with externally evolved schema - #4140
Merged
Conversation
Document the known issue (apache#4038) where external schema changes on a Fluss-managed Paimon table stall the tiering service. Closes apache#4138.
gstamatakis95
force-pushed
the
fix-4138
branch
from
August 28, 2026 09:13
0142431 to
1659ef6
Compare
gstamatakis95
marked this pull request as ready for review
August 28, 2026 09:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Closes #4138
Documents the known issue reported in #4038. When the schema of a Paimon lake table managed by Fluss is evolved by an external engine (for example Spark or Doris adding a column directly on the Paimon table), the tiering job fails to write subsequent records and restarts in a loop until the schemas match again.
There is no code fix for this yet, so this PR documents the failure, its blast radius, the safety guarantees, and the recovery steps, and states the operational rule that lake table schemas must only be evolved through Fluss.
Summary
website/docs/streaming-lakehouse/datalake-formats/paimon.md: adds a Schema Evolution section describing the supported evolution path (ALTER TABLE ... ADDon the Fluss table, which applies the same change to the Paimon table as part of the statement) and a warning covering the external evolution failure: the exact error signature, the job-level restart loop visible in the Flink UI, the impact on the other tables served by the same tiering job, the paused log retention and growing storage while tiering is stalled, and the recovery steps (run the matchingALTER TABLE ... ADDon the Fluss table to keep the column, or drop the externally added column when Fluss cannot reconcile the schemas).website/docs/streaming-lakehouse/tiering-service.md: adds a warning in the Overview that the schema of a lake table managed by the Tiering Service must never be changed through an external engine, notes that only Paimon and Iceberg support schema evolution through Fluss, and links the Paimon Schema Evolution section for symptoms and recovery.website/docs/engine-flink/ddl.md: adds a note in the Add Columns section that datalake-enabled tables on a Paimon or Iceberg lakehouse propagate added columns to the corresponding lake table, that adding columns to datalake-enabled Hudi and Lance tables is not yet supported, and that the lake table must not be altered externally, linking the Paimon Schema Evolution section.🤖 AI-assisted changes - reviewed by human developer