Skip to content

Core: Add newTableMetadata option to preserve existing field IDs - #18584

Closed
hreddygayam wants to merge 1 commit into
apache:mainfrom
hreddygayam:preserve-field-ids-on-create
Closed

hreddygayam wants to merge 1 commit into
apache:mainfrom
hreddygayam:preserve-field-ids-on-create

Conversation

@hreddygayam

Copy link
Copy Markdown
Contributor

What

TableMetadata.newTableMetadata(...) always reassigns fresh, sequential field IDs via TypeUtil.assignFreshIds, so there is currently no supported way to create table metadata that keeps the field IDs already set on the provided schema. This matters when the IDs are already meaningful — for example migrating or re-registering a table whose data files were written against a specific ID assignment.

This adds a public overload with a reassignFieldIds flag:

public static TableMetadata newTableMetadata(
    Schema schema, PartitionSpec spec, SortOrder sortOrder,
    String location, Map<String, String> properties, boolean reassignFieldIds)
  • reassignFieldIds = true — what every existing overload now delegates to — keeps today's behavior exactly.
  • reassignFieldIds = false preserves the schema's field IDs: the schema is rebuilt with the initial schema ID, last-column-id is taken from schema.highestFieldId() so later column additions don't collide, and the IDs are validated to be positive (uniqueness is already guaranteed by Schema construction). The partition spec and sort order are still rebuilt against the resulting schema, so their source IDs are preserved as well.

Why

Closes #17614.

As discussed on the issue, this is intentionally a narrow, expert-facing method rather than a Catalog.TableBuilder option — it avoids a flag that other catalog implementations (REST client, engine catalogs) would silently ignore, and avoids a REST OpenAPI CreateTableRequest change. A TableBuilder option can follow in a separate PR if there's appetite.

cc @RussellSpitzer — this is the "new method for experts" shape you suggested on the issue; does the signature look right before I invest further?

Tests

Added to TestTableMetadata:

  • preserves field IDs (including nested fields and a partition source column),
  • reassignFieldIds = true matches the existing fresh, sequential ID assignment,
  • non-positive field IDs are rejected.

:iceberg-core:test, :iceberg-core:checkstyleMain, and :iceberg-core:revapi all pass (the new public static method is a source- and binary-compatible addition).

TableMetadata.newTableMetadata always reassigns fresh, sequential field IDs,
so there is no way to create table metadata while keeping the field IDs that
are already meaningful on the provided schema (e.g. when migrating or
re-registering a table whose data files were written against a specific ID
assignment).

Add a public newTableMetadata overload with a reassignFieldIds flag. When
false, the schema's field IDs are preserved (validated to be positive), the
schema ID is reset to the initial ID for a new table, and last-column-id is
taken from the schema's highest field ID so later column additions don't
collide. The partition spec and sort order are still rebuilt against the
resulting schema, so their source IDs are preserved as well. Existing
overloads delegate with reassignFieldIds=true, leaving default behavior
unchanged.

Closes apache#17614
@hreddygayam

Copy link
Copy Markdown
Contributor Author

Closing for now — I opened this a bit prematurely. I'll re-open once it's gone through review on my side. Apologies for the noise.

@hreddygayam
hreddygayam deleted the preserve-field-ids-on-create branch October 19, 2026 10:06
@github-actions github-actions Bot added the core label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support preserving field IDs when creating a table from an existing schema

1 participant