Repository navigation
Spec: Add V4 column updates to the spec - #16425
anuragmantri wants to merge 1 commit into
Conversation
github-actions
Bot
added
the
Specification
d997ce3 to
7f300b3
Compare
|
|
||
| * Rows in a column file with a null `_last_updated_sequence_number` inherit the base data file's (bumped) data sequence number. This applies to rows whose values were updated in the commit that produced the column file. | ||
| * Rows that carry over unchanged values from a prior column file retain their original `_last_updated_sequence_number`, which the writer materializes as a physical column in the column file. | ||
| * The `_row_id` for updated rows is unchanged; it is derived from the base data file's `first_row_id` and the row's position. Column files do not reassign row ids. |
There was a problem hiding this comment.
This is confusing to me, it assumes that the _row_id value isn't materialized yet (and can therefore be derived from the "base" data_file), but I don't understand how we can make that assumption?
Wait.. I just realized, the _row_id of a data file should not be in a column-data-file.
I finally understand this section, but it's not worded very clearly.
I think it should instead say:
Column data files MUST NOT contain the _row_id field (field-id 2147483540). Regular "Row ID Assignment"(link to section) rules for reading apply.
There was a problem hiding this comment.
Thanks, agreed. The section now says column files must not contain the reserved _row_id (or _pos) fields, and that adding column files doesn't change row IDs, so the regular Row Lineage rules apply. The old "derived from the base data file" wording is gone.
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
e9e3ea8 to
850bb36
Compare
850bb36 to
89f73c3
Compare
anuragmantri
left a comment
anuragmantri
left a comment
There was a problem hiding this comment.
I rebased this on main now that #16025 is merged and rewrote it as the column update rules.
|
|
||
| * Rows in a column file with a null `_last_updated_sequence_number` inherit the base data file's (bumped) data sequence number. This applies to rows whose values were updated in the commit that produced the column file. | ||
| * Rows that carry over unchanged values from a prior column file retain their original `_last_updated_sequence_number`, which the writer materializes as a physical column in the column file. | ||
| * The `_row_id` for updated rows is unchanged; it is derived from the base data file's `first_row_id` and the row's position. Column files do not reassign row ids. |
There was a problem hiding this comment.
Thanks, agreed. The section now says column files must not contain the reserved _row_id (or _pos) fields, and that adding column files doesn't change row IDs, so the regular Row Lineage rules apply. The old "derived from the base data file" wording is gone.
|
|
||
| - `tracking`: | ||
| - `tracking.status` is MODIFIED. | ||
| - `tracking.snapshot_id` must not be null. |
There was a problem hiding this comment.
| - `tracking.snapshot_id` must not be null. | |
| - `tracking.snapshot_id` must be the previous entry's (inherited) snapshot ID. |
| - `column_files`: | ||
| - New column files are added to the previous entry's `column_files`. | ||
| - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. | ||
| - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and must be the row's current `_last_updated_sequence_number`, read from the previous entry, for every other row that is not deleted. |
There was a problem hiding this comment.
| - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and must be the row's current `_last_updated_sequence_number`, read from the previous entry, for every other row that is not deleted. | |
| - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and must be the row's current `_last_updated_sequence_number`, the non-null value a reader of the previous entry would see, for every other row that is not deleted. |
There was a problem hiding this comment.
Shall we do this with the _last_update_time too?
| - `sort_order_id`: | ||
| - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. | ||
| - `content_stats`: | ||
| - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. |
There was a problem hiding this comment.
Do we want to talk about the stats of deleted rows?
| - `content_stats`: | ||
| - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. | ||
| - `deletion_vector`: | ||
| - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. |
There was a problem hiding this comment.
Is it discussed how a change data reader will detect that these rows were deleted previously, and they are not new deletes?
|
|
||
| Starting in v4, the values of one or more fields in a data file can be updated without rewriting the data file by adding column files using the [`column_files`](#column-file-fields) field. This section describes column files for data files. | ||
|
|
||
| In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. |
There was a problem hiding this comment.
| In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row modified by the column update. | |
| In this section, the **previous entry** is the live entry of the data file before a column update, the **MODIFIED entry** is the entry produced by the column update, an **updated field** is a field whose ID is in the `field_ids` of a column file added by the column update, and an **updated row** is a row whose updated-field values were set by the update; other rows keep their current values. |
gaborkaszab
left a comment
gaborkaszab
left a comment
There was a problem hiding this comment.
Thanks for the PR, @anuragmantri !
Great work! I've made my first pass now to get familiar.
| * **Delete manifest** -- A file that lists delete files to be associated with data files at planning time. | ||
| * **Data file** -- A file that contains rows of a table. | ||
| * **Delete file** -- A file that encodes rows of a table that are deleted by position or data values. | ||
| * **Column file** -- (v4+) A file that stores the values of one or more fields for each row of a data file, or for each entry of a v4 leaf manifest. |
There was a problem hiding this comment.
nit: for each row of and for each entry of seems an implementation detail that we decided on the dense representation. Might not be relevant here when we define column files.
Suggestion:
A file that stores the values of one or more fields of a data file, or a v4 leaf manifest ?
|
|
||
| Table state is maintained in metadata files. All changes to table state create a new metadata file and replace the old metadata with an atomic swap. The table metadata file tracks the table schema, partitioning config, custom properties, and snapshots of the table contents. A snapshot represents the state of a table at some time and is used to access the complete set of data files in the table. | ||
|
|
||
| Data files in snapshots are tracked by one or more manifest files that contain a row for each data file in the table, the file's partition data, an optional colocated deletion vector (v4), and its metrics. The data in a snapshot is the union of all live files in its manifests; each live file may only appear once (see [Content file uniqueness](#content-file-uniqueness)). Manifest files are reused across snapshots to avoid rewriting metadata that is slow-changing. Manifests can track data files with any subset of a table and are not associated with partitions. |
There was a problem hiding this comment.
Since we mention V4 and colocated deletion vectors already, shouldn't we also mention column files too?
|
|
||
| Version 4 of the Iceberg spec restructures metadata for improved performance and new capabilities: | ||
|
|
||
| * Support for an [adaptive metadata tree](#manifests), enabling efficient small commits, column updates, and columnar statistics representation |
There was a problem hiding this comment.
More general column files instead of column updates?
|
|
||
| When `null`, a row's `_row_id` field is assigned to the `first_row_id` from its containing data file plus the row position in that data file (`_pos`). A data file's `first_row_id` field is assigned using inheritance and is documented in [First Row ID Inheritance](#first-row-id-inheritance). A manifest's `first_row_id` is assigned when writing the snapshot root file and is documented in [Snapshot Root File First Row ID Assignment](#snapshot-root-file-first-row-id-assignment). Starting in v4, a data file in the root manifest is assigned a `first_row_id` in the same way. A snapshot's `first-row-id` is set to the table's `next-row-id` and is documented in [Snapshot Row IDs](#snapshot-row-ids). | ||
|
|
||
| When an existing row is moved to a different data file for any reason, writers should write `_row_id` and `_last_updated_sequence_number` according to the following rules: |
There was a problem hiding this comment.
This section explains what is expected when we write an entire row into a new file. Shouldn't we also cover the column update case here, mentioning that _row_id is not expected in the column file, but _lusn is a must?
I see a section below that covers this in the context of writing column files, but I believe that the "Row Lineage" section is more self-contained with covering column updates too.
| ### Partitioning | ||
|
|
||
| Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. Manifests store data files from any partition. Manifests in v3 and earlier store files for a single spec. Starting in v4, manifests may store files from any spec. A manifest is considered partitioned by a spec if every entry in the manifest is partitioned by that spec. | ||
| Data files are stored in manifests with a tuple of partition values that are used in scans to filter out files that cannot contain records that match the scan’s filter predicate. Partition values for a data file must be the same for all records stored in the data file. For a data file with column files, this requirement applies only to rows that are not deleted, using values read with its column files. Manifests store data files from any partition. Manifests in v3 and earlier store files for a single spec. Starting in v4, manifests may store files from any spec. A manifest is considered partitioned by a spec if every entry in the manifest is partitioned by that spec. |
There was a problem hiding this comment.
Maybe it's just me, but I don't get the point of this sentence:
For a data file with column files, this requirement applies only to rows that are not deleted, using values read with its column files.
Do you say that the invariant of all records in a data file should have the same partition values is not true when we have column files on the data file?
| - If an updated field is, or contains, the source of a field in the previous entry's partition spec, `partition` must be the partition tuple shared by the rows that are not deleted, or `spec_id` and `partition` must be null. | ||
| - If `spec_id` and `partition` are set to null, each source field of an identity partition field in the previous entry's partition spec that is missing from the data file must be stored in a column file of the MODIFIED entry that has the source's top-level field ID in its `field_ids`. | ||
| - `sort_order_id`: | ||
| - If an updated field is, or contains, the source of a field in the previous entry's sort order, `sort_order_id` must be null. |
There was a problem hiding this comment.
I find these explanations with "previous entry" and such a bit hard to read. Can't just simply say If we update one or more field that are part of the sort order fields, then sort_order_id must be set to null ?
| - `content_stats`: | ||
| - Stats for each updated field, including its nested fields, must be computed from the values in the column file that has its ID in `field_ids`, or omitted. Stats for other fields may be kept. | ||
| - `deletion_vector`: | ||
| - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. |
There was a problem hiding this comment.
Can't we rephrase this to be simpler, like existing deletes on the data file should be written into deletion_vector including equality deletes ? Or something similar :)
| - `deletion_vector`: | ||
| - `deletion_vector` must contain all deletes from the previous entry's `deletion_vector` and from delete files and deletion vectors in delete manifests that apply to the previous entry, including equality delete files. | ||
| - `column_files`: | ||
| - New column files are added to the previous entry's `column_files`. |
There was a problem hiding this comment.
This sentence seems misleading, gives me the impression that we add the new column files into the previous entry, but we rather add them to the new entry.
| - `column_files`: | ||
| - New column files are added to the previous entry's `column_files`. | ||
| - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. | ||
| - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and must be the row's current `_last_updated_sequence_number`, read from the previous entry, for every other row that is not deleted. |
There was a problem hiding this comment.
"for every other row that is not deleted." I wouldn't mention deleted rows here, we don't really have ay assumptions to their _LUSN value. Maybe a simplified version:
Its value must be null for each updated row, and for the non-updated rows it must be the _last_updated_sequence_number read from the previous entry
| - Each updated field ID must be removed from the `field_ids` of the previous entry's column files, and a column file with no remaining `field_ids` must be removed. | ||
| - One of the new column files must have the `_last_updated_sequence_number` field ID in its `field_ids`. Its value must be null for each updated row, and must be the row's current `_last_updated_sequence_number`, read from the previous entry, for every other row that is not deleted. | ||
|
|
||
| Deletion vectors in delete manifests that reference the data file must be removed (see [Deletion Vectors](#deletion-vectors)). After column files are added, delete files in delete manifests do not apply to the data file (see [Scan Planning](#scan-planning)). |
There was a problem hiding this comment.
Is this true? Don't we guarantee this by bumping the data sequence number?
This branch has not been deployed
5 participants
These are the spec changes for V4 column updates from this proposal. This will be in conjunction with other V4 spec PRs #16025, #14234 and #15630