Repository navigation
Core: Add newTableMetadata option to preserve existing field IDs - #18584
Closed
hreddygayam wants to merge 1 commit into
Closed
hreddygayam wants to merge 1 commit into
hreddygayam wants to merge 1 commit into
Conversation
TableMetadata.newTableMetadata always reassigns fresh, sequential field IDs, so there is no way to create table metadata while keeping the field IDs that are already meaningful on the provided schema (e.g. when migrating or re-registering a table whose data files were written against a specific ID assignment). Add a public newTableMetadata overload with a reassignFieldIds flag. When false, the schema's field IDs are preserved (validated to be positive), the schema ID is reset to the initial ID for a new table, and last-column-id is taken from the schema's highest field ID so later column additions don't collide. The partition spec and sort order are still rebuilt against the resulting schema, so their source IDs are preserved as well. Existing overloads delegate with reassignFieldIds=true, leaving default behavior unchanged. Closes apache#17614
Contributor
Author
|
Closing for now — I opened this a bit prematurely. I'll re-open once it's gone through review on my side. Apologies for the noise. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
TableMetadata.newTableMetadata(...)always reassigns fresh, sequential field IDs viaTypeUtil.assignFreshIds, so there is currently no supported way to create table metadata that keeps the field IDs already set on the provided schema. This matters when the IDs are already meaningful — for example migrating or re-registering a table whose data files were written against a specific ID assignment.This adds a public overload with a
reassignFieldIdsflag:reassignFieldIds = true— what every existing overload now delegates to — keeps today's behavior exactly.reassignFieldIds = falsepreserves the schema's field IDs: the schema is rebuilt with the initial schema ID,last-column-idis taken fromschema.highestFieldId()so later column additions don't collide, and the IDs are validated to be positive (uniqueness is already guaranteed bySchemaconstruction). The partition spec and sort order are still rebuilt against the resulting schema, so their source IDs are preserved as well.Why
Closes #17614.
As discussed on the issue, this is intentionally a narrow, expert-facing method rather than a
Catalog.TableBuilderoption — it avoids a flag that other catalog implementations (REST client, engine catalogs) would silently ignore, and avoids a REST OpenAPICreateTableRequestchange. ATableBuilderoption can follow in a separate PR if there's appetite.cc @RussellSpitzer — this is the "new method for experts" shape you suggested on the issue; does the signature look right before I invest further?
Tests
Added to
TestTableMetadata:reassignFieldIds = truematches the existing fresh, sequential ID assignment,:iceberg-core:test,:iceberg-core:checkstyleMain, and:iceberg-core:revapiall pass (the new public static method is a source- and binary-compatible addition).