From 2b404ad5a0c3312bd074d502a68b17d5988e07a9 Mon Sep 17 00:00:00 2001 From: Anoop Johnson Date: Mon, 19 Oct 2026 10:49:29 -0700 Subject: [PATCH 01/10] Spec: Replace per-column tight_bounds with file-level tight_stats in v4 Track stats tightness with a single file-level tight_stats flag on the tracked file instead of a per-column tight_bounds field. The flag covers all statistics (bounds and null, NaN, and value counts) for the file's live rows. --- format/spec.md | 23 ++++++++++++++++++----- 1 file changed, 18 insertions(+), 5 deletions(-) diff --git a/format/spec.md b/format/spec.md index 3e8ba7232f72..1ee5175d77b2 100644 --- a/format/spec.md +++ b/format/spec.md @@ -785,6 +785,7 @@ The contents of a file are part of a snapshot if its tracking `status` is **live | _optional_ | 102 | **`partition`** | `struct<...>` | Partition data tuple for the file; null if unpartitioned. | | _optional_ | 140 | **`sort_order_id`** | `int` | ID representing sort order for this file. If missing or unknown, the order is assumed to be unsorted. | | _optional_ | 146 | **`content_stats`** | `content_stats` struct | Field-level stats. See [Content Stats](#content-stats). | + | _optional_ | 151 | **`tight_stats`** | `boolean` | When true, the file's `content_stats` are exact for its live rows. See [Stats Tightness](#stats-tightness). | | _optional_ | 150 | **`manifest_info`** | `manifest_info` struct | Manifest-specific stats. See [Manifest Info](#manifest-info-fields). | | _optional_ | 148 | **`deletion_vector`** | `deletion_vector` struct | Row-level deletion vector for a data file. See [Deletion Vector](#deletion-vector-fields). | | _optional_ | 158 | **`column_files`** | `list<159: column_file>` | Column files associated with this file. See [Column File](#column-file-fields). | @@ -868,6 +869,8 @@ The contents of a file are part of a snapshot if its tracking `status` is **live - `column_files`: - `column_files` may only be set if the tracked file is a data file or a v4 leaf manifest. - A field ID may appear in `field_ids` of at most one column file in a tracked file's `column_files`. + - `tight_stats`: + - `tight_stats` may be true only if `content_stats` are exact for the file's live rows; setting or updating a `deletion_vector` without recomputing stats makes them inexact. When a file is added to the dataset, its tracked file must set status to ADDED and store the snapshot ID in which the file was added. @@ -949,7 +952,6 @@ Each stats struct holds statistics for one table field. It may contain the follo |-------------|--------|---------------------------|---------------------------|-----------------------------------------------|-------------| | _optional_ | 1 | `lower_bound` | Field type or `geo_lower` | all primitives or `variant` | Lower bound stored as the field's type, or `geo_lower` for geo types | | _optional_ | 2 | `upper_bound` | Field type or `geo_upper` | all primitives or `variant` | Upper bound stored as the field's type, or `geo_upper` for geo types | -| _optional_ | 3 | `tight_bounds` | `boolean` | all primitives except for `geometry` and `geography` | When true, `lower_bound` and `upper_bound` must be equal to the min and max values | | _optional_ | 4 | `value_count` | `long` | all | Number of values in the column (including null and NaN values) | | _optional_ | 5 | `null_value_count` | `long` | optional fields | Number of null values in the column | | _optional_ | 6 | `nan_value_count` | `long` | `float`, `double` | Number of NaN values in the column | @@ -961,7 +963,6 @@ For example, stats for a `required` `int` field named `id` with field-id `2` are 10_400: optional struct id (default null) { 10_401: optional int lower_bound; // type matches the field type (int) 10_402: optional int upper_bound; // type matches the field type (int) - 10_403: optional boolean tight_bounds; 10_404: optional long value_count; // null_value_count is only used for optional fields @@ -1011,12 +1012,26 @@ For example, stats for an optional `geometry` field named `location` with field- 10_804: optional long value_count; 10_805: optional long null_value_count; 10_807: optional long total_bytes; - // tight_bounds and nan_value_count are omitted for geo types + // nan_value_count is omitted for geo types } ``` For `variant`, both bounds are unshredded `variant` that store variant field bounds by normalized JSON paths as field names. See [Bounds for Variant](#bounds-for-variant) for details on producing these bounds. + +###### Stats Tightness + +Statistics are *tight* when they are exact for the live rows of a file: bounds are the minimum and maximum of the live values and the counts (`value_count`, `null_value_count`, and `nan_value_count`) are exact. When `tight_stats` is absent, readers must assume stats are not tight. + +When `tight_stats` is true: + +* Counts are exact for all fields. +* Bounds are exact for fixed-width primitives and for `variant`. +* Bounds for `string` and `binary` are not exact because they may have been truncated when written. +* `geometry` and `geography` bounds are a bounding box that contains the live values, not exact minimum and maximum values. + +A deletion vector does not by itself make a file's stats non-tight; `tight_stats` is the only indicator of tightness. Stats may be recomputed to restore tightness without rewriting the data file. + ###### Content Stats in Manifests Manifest files are written using a specific `content_stats` container struct type, determined by the writer and incorporated into the manifest schema. All field-level structs are optional fields in the `content_stats` struct. @@ -1029,7 +1044,6 @@ For example, stats for a table with a required int, `id`, and an optional string 10_400: optional struct id (default null) { 10_401: optional int lower_bound; 10_402: optional int upper_bound; - 10_403: optional boolean tight_bounds; 10_404: optional long value_count; } @@ -1037,7 +1051,6 @@ For example, stats for a table with a required int, `id`, and an optional string 10_600: optional struct data (default null) { 10_601: optional string lower_bound; 10_602: optional string upper_bound; - 10_603: optional boolean tight_bounds; 10_604: optional long value_count; 10_605: optional long null_value_count; 10_607: optional long total_bytes; From 44294ca353b1d74bcdeed8711a7fe8fa478b07bd Mon Sep 17 00:00:00 2001 From: Anoop Johnson Date: Mon, 19 Oct 2026 13:15:30 -0700 Subject: [PATCH 02/10] Spec: Clarify tight_stats bound exactness for variant and geo types --- format/spec.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/format/spec.md b/format/spec.md index 1ee5175d77b2..81e0b74c473b 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1026,9 +1026,8 @@ Statistics are *tight* when they are exact for the live rows of a file: bounds a When `tight_stats` is true: * Counts are exact for all fields. -* Bounds are exact for fixed-width primitives and for `variant`. -* Bounds for `string` and `binary` are not exact because they may have been truncated when written. -* `geometry` and `geography` bounds are a bounding box that contains the live values, not exact minimum and maximum values. +* Bounds are exact for primitive types other than `string` and `binary`, whose bounds may have been truncated when written. +* `variant`, `geometry`, and `geography` do not have exact bounds. A deletion vector does not by itself make a file's stats non-tight; `tight_stats` is the only indicator of tightness. Stats may be recomputed to restore tightness without rewriting the data file. From 5d579635a6b9e9e2130daaedb34ce99e9d24c32a Mon Sep 17 00:00:00 2001 From: Anoop Johnson Date: Mon, 19 Oct 2026 13:34:44 -0700 Subject: [PATCH 03/10] clarify variant and geometry --- format/spec.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index 81e0b74c473b..8cd7d6e09df6 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1027,7 +1027,8 @@ When `tight_stats` is true: * Counts are exact for all fields. * Bounds are exact for primitive types other than `string` and `binary`, whose bounds may have been truncated when written. -* `variant`, `geometry`, and `geography` do not have exact bounds. +* `variant` bounds are stored per leaf field and are optional; a leaf field may have no bound, but any bound that is present is exact. +* `geometry` and `geography` bounds are a bounding box that contains the live values, not exact minimum and maximum values. A deletion vector does not by itself make a file's stats non-tight; `tight_stats` is the only indicator of tightness. Stats may be recomputed to restore tightness without rewriting the data file. From 448217d6fd307709887ad5500801a0ad7126aa3e Mon Sep 17 00:00:00 2001 From: Anoop Johnson Date: Mon, 19 Oct 2026 15:39:08 -0700 Subject: [PATCH 04/10] Spec: Refine v4 stats tightness for counts --- format/spec.md | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/format/spec.md b/format/spec.md index 8cd7d6e09df6..c154dbf3d5d9 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1021,15 +1021,24 @@ For `variant`, both bounds are unshredded `variant` that store variant field bou ###### Stats Tightness -Statistics are *tight* when they are exact for the live rows of a file: bounds are the minimum and maximum of the live values and the counts (`value_count`, `null_value_count`, and `nan_value_count`) are exact. When `tight_stats` is absent, readers must assume stats are not tight. +Statistics are *tight* when they are exact for the live rows of a file: + +* Bounds are the minimum and maximum of the live values. +* `null_value_count` and `nan_value_count` are the counts across the live rows. + +`value_count` and `total_bytes` are not part of tightness. `value_count` is the number of values across all rows written to the file and is not adjusted when rows are deleted; for a field with no repeated (`list` or `map`) ancestor, the live value count is `value_count` minus the deletion vector's cardinality. `total_bytes` is an approximate optimizer input and is likewise not adjusted for deleted rows. + +When `tight_stats` is absent, readers must assume stats are not tight. When `tight_stats` is true: -* Counts are exact for all fields. +* `null_value_count` and `nan_value_count` are exact for the live rows. * Bounds are exact for primitive types other than `string` and `binary`, whose bounds may have been truncated when written. * `variant` bounds are stored per leaf field and are optional; a leaf field may have no bound, but any bound that is present is exact. * `geometry` and `geography` bounds are a bounding box that contains the live values, not exact minimum and maximum values. +A `variant` value is null only when the variant itself is null; a non-null variant that encodes a variant null is not counted as a null value. Variants do not have a `nan_value_count`. + A deletion vector does not by itself make a file's stats non-tight; `tight_stats` is the only indicator of tightness. Stats may be recomputed to restore tightness without rewriting the data file. ###### Content Stats in Manifests From 61f0eb430f2459843790d07da2f35b29a0e985e0 Mon Sep 17 00:00:00 2001 From: Anoop Johnson Date: Mon, 19 Oct 2026 16:04:26 -0700 Subject: [PATCH 05/10] Drop the part about live rows --- format/spec.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/format/spec.md b/format/spec.md index c154dbf3d5d9..1fd8e7f0c89d 100644 --- a/format/spec.md +++ b/format/spec.md @@ -785,7 +785,7 @@ The contents of a file are part of a snapshot if its tracking `status` is **live | _optional_ | 102 | **`partition`** | `struct<...>` | Partition data tuple for the file; null if unpartitioned. | | _optional_ | 140 | **`sort_order_id`** | `int` | ID representing sort order for this file. If missing or unknown, the order is assumed to be unsorted. | | _optional_ | 146 | **`content_stats`** | `content_stats` struct | Field-level stats. See [Content Stats](#content-stats). | - | _optional_ | 151 | **`tight_stats`** | `boolean` | When true, the file's `content_stats` are exact for its live rows. See [Stats Tightness](#stats-tightness). | + | _optional_ | 151 | **`tight_stats`** | `boolean` | When true, the file's `content_stats` are tight. See [Stats Tightness](#stats-tightness). | | _optional_ | 150 | **`manifest_info`** | `manifest_info` struct | Manifest-specific stats. See [Manifest Info](#manifest-info-fields). | | _optional_ | 148 | **`deletion_vector`** | `deletion_vector` struct | Row-level deletion vector for a data file. See [Deletion Vector](#deletion-vector-fields). | | _optional_ | 158 | **`column_files`** | `list<159: column_file>` | Column files associated with this file. See [Column File](#column-file-fields). | @@ -870,7 +870,7 @@ The contents of a file are part of a snapshot if its tracking `status` is **live - `column_files` may only be set if the tracked file is a data file or a v4 leaf manifest. - A field ID may appear in `field_ids` of at most one column file in a tracked file's `column_files`. - `tight_stats`: - - `tight_stats` may be true only if `content_stats` are exact for the file's live rows; setting or updating a `deletion_vector` without recomputing stats makes them inexact. + - `tight_stats` may be true only if `content_stats` are tight; attaching or updating a `deletion_vector` without recomputing stats makes them no longer tight. See [Stats Tightness](#stats-tightness). When a file is added to the dataset, its tracked file must set status to ADDED and store the snapshot ID in which the file was added. From b9675a7c5f94c9827d6fce9a9f5cab99e03fe8f4 Mon Sep 17 00:00:00 2001 From: Anoop Johnson Date: Mon, 19 Oct 2026 16:08:29 -0700 Subject: [PATCH 06/10] Fix total_bytes wording --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index 1fd8e7f0c89d..89de87a7798b 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1026,7 +1026,7 @@ Statistics are *tight* when they are exact for the live rows of a file: * Bounds are the minimum and maximum of the live values. * `null_value_count` and `nan_value_count` are the counts across the live rows. -`value_count` and `total_bytes` are not part of tightness. `value_count` is the number of values across all rows written to the file and is not adjusted when rows are deleted; for a field with no repeated (`list` or `map`) ancestor, the live value count is `value_count` minus the deletion vector's cardinality. `total_bytes` is an approximate optimizer input and is likewise not adjusted for deleted rows. +`value_count` and `total_bytes` are not part of tightness. `value_count` is the number of values across all rows written to the file and is not adjusted when rows are deleted; for a field with no repeated (`list` or `map`) ancestor, the live value count is `value_count` minus the deletion vector's cardinality. `total_bytes` is an estimate and is likewise not adjusted for deleted rows. When `tight_stats` is absent, readers must assume stats are not tight. From 1bbf989a2ac271bf0c40c970a51379b8fa8f4cea Mon Sep 17 00:00:00 2001 From: Anoop Johnson Date: Tue, 20 Oct 2026 09:33:47 -0700 Subject: [PATCH 07/10] Spec: Address review feedback on stats tightness --- format/spec.md | 23 ++++++----------------- 1 file changed, 6 insertions(+), 17 deletions(-) diff --git a/format/spec.md b/format/spec.md index 89de87a7798b..83596fe33285 100644 --- a/format/spec.md +++ b/format/spec.md @@ -870,7 +870,7 @@ The contents of a file are part of a snapshot if its tracking `status` is **live - `column_files` may only be set if the tracked file is a data file or a v4 leaf manifest. - A field ID may appear in `field_ids` of at most one column file in a tracked file's `column_files`. - `tight_stats`: - - `tight_stats` may be true only if `content_stats` are tight; attaching or updating a `deletion_vector` without recomputing stats makes them no longer tight. See [Stats Tightness](#stats-tightness). + - `tight_stats` may be true only if `content_stats` are tight. See [Stats Tightness](#stats-tightness). When a file is added to the dataset, its tracked file must set status to ADDED and store the snapshot ID in which the file was added. @@ -1018,28 +1018,17 @@ For example, stats for an optional `geometry` field named `location` with field- For `variant`, both bounds are unshredded `variant` that store variant field bounds by normalized JSON paths as field names. See [Bounds for Variant](#bounds-for-variant) for details on producing these bounds. - ###### Stats Tightness -Statistics are *tight* when they are exact for the live rows of a file: +Statistics are *tight* when the bounds, `null_value_count`, and `nan_value_count` are exact for the live rows of a file: -* Bounds are the minimum and maximum of the live values. -* `null_value_count` and `nan_value_count` are the counts across the live rows. - -`value_count` and `total_bytes` are not part of tightness. `value_count` is the number of values across all rows written to the file and is not adjusted when rows are deleted; for a field with no repeated (`list` or `map`) ancestor, the live value count is `value_count` minus the deletion vector's cardinality. `total_bytes` is an estimate and is likewise not adjusted for deleted rows. - -When `tight_stats` is absent, readers must assume stats are not tight. - -When `tight_stats` is true: - -* `null_value_count` and `nan_value_count` are exact for the live rows. -* Bounds are exact for primitive types other than `string` and `binary`, whose bounds may have been truncated when written. +* Bounds are the minimum and maximum of the live values. Bounds for `string` and `binary` are never tight because they may be truncated when written, and bounds for `geometry` and `geography` are a bounding box, not exact minimum and maximum values. * `variant` bounds are stored per leaf field and are optional; a leaf field may have no bound, but any bound that is present is exact. -* `geometry` and `geography` bounds are a bounding box that contains the live values, not exact minimum and maximum values. +* `null_value_count` and `nan_value_count` are the counts across the live rows. A `variant` value is null only when the variant itself is null; a non-null variant that encodes a variant null is not counted as a null value. -A `variant` value is null only when the variant itself is null; a non-null variant that encodes a variant null is not counted as a null value. Variants do not have a `nan_value_count`. +Statistics other than the bounds, `null_value_count`, and `nan_value_count` are not part of tightness. In particular, `value_count` is the number of values across all rows written to the file and is not adjusted when rows are deleted; for a field with no repeated (`list` or `map`) ancestor, the live value count is `value_count` minus the deletion vector's cardinality. -A deletion vector does not by itself make a file's stats non-tight; `tight_stats` is the only indicator of tightness. Stats may be recomputed to restore tightness without rewriting the data file. +When `tight_stats` is absent, readers must assume stats are not tight. A deletion vector does not by itself make a file's stats non-tight; `tight_stats` is the only indicator of tightness. Stats may be recomputed to restore tightness without rewriting the data file. ###### Content Stats in Manifests From 8121b2abf5f12c5285d4d58b1f56f88bd92bfb35 Mon Sep 17 00:00:00 2001 From: Anoop Johnson Date: Tue, 20 Oct 2026 12:00:56 -0700 Subject: [PATCH 08/10] PR feedback --- format/spec.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/format/spec.md b/format/spec.md index 83596fe33285..70c14a6fe335 100644 --- a/format/spec.md +++ b/format/spec.md @@ -869,8 +869,6 @@ The contents of a file are part of a snapshot if its tracking `status` is **live - `column_files`: - `column_files` may only be set if the tracked file is a data file or a v4 leaf manifest. - A field ID may appear in `field_ids` of at most one column file in a tracked file's `column_files`. - - `tight_stats`: - - `tight_stats` may be true only if `content_stats` are tight. See [Stats Tightness](#stats-tightness). When a file is added to the dataset, its tracked file must set status to ADDED and store the snapshot ID in which the file was added. @@ -1020,11 +1018,14 @@ For `variant`, both bounds are unshredded `variant` that store variant field bou ###### Stats Tightness -Statistics are *tight* when the bounds, `null_value_count`, and `nan_value_count` are exact for the live rows of a file: +Statistics are _tight_ when the bounds, `null_value_count`, and `nan_value_count` are exact for the live rows of a file: -* Bounds are the minimum and maximum of the live values. Bounds for `string` and `binary` are never tight because they may be truncated when written, and bounds for `geometry` and `geography` are a bounding box, not exact minimum and maximum values. -* `variant` bounds are stored per leaf field and are optional; a leaf field may have no bound, but any bound that is present is exact. -* `null_value_count` and `nan_value_count` are the counts across the live rows. A `variant` value is null only when the variant itself is null; a non-null variant that encodes a variant null is not counted as a null value. +* `lower_bound` and `upper_bound` for fixed-width types are equal to the minimum and maximum of the live values. +* `null_value_count` and `nan_value_count` are the counts across the live rows. + +Bounds for `string` and `binary` are never tight because they may be truncated when written. Bounds for `geometry` and `geography` are a bounding box, not exact minimum and maximum values. + +`variant` bounds are stored per leaf field and are optional; a leaf field may have no bound, but any bound that is present is exact. A `variant` value is null only when the variant itself is null; a non-null variant that encodes a variant null is not counted as a null value. Statistics other than the bounds, `null_value_count`, and `nan_value_count` are not part of tightness. In particular, `value_count` is the number of values across all rows written to the file and is not adjusted when rows are deleted; for a field with no repeated (`list` or `map`) ancestor, the live value count is `value_count` minus the deletion vector's cardinality. From 1028d26c258e265b32840b55a26feedce1a1bda5 Mon Sep 17 00:00:00 2001 From: Anoop Johnson Date: Tue, 20 Oct 2026 13:10:41 -0700 Subject: [PATCH 09/10] Feedback --- format/spec.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/format/spec.md b/format/spec.md index 70c14a6fe335..b27818c8076f 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1020,7 +1020,7 @@ For `variant`, both bounds are unshredded `variant` that store variant field bou Statistics are _tight_ when the bounds, `null_value_count`, and `nan_value_count` are exact for the live rows of a file: -* `lower_bound` and `upper_bound` for fixed-width types are equal to the minimum and maximum of the live values. +* `lower_bound` and `upper_bound` for fixed-length types are equal to the minimum and maximum of the live values. * `null_value_count` and `nan_value_count` are the counts across the live rows. Bounds for `string` and `binary` are never tight because they may be truncated when written. Bounds for `geometry` and `geography` are a bounding box, not exact minimum and maximum values. @@ -1029,7 +1029,7 @@ Bounds for `string` and `binary` are never tight because they may be truncated w Statistics other than the bounds, `null_value_count`, and `nan_value_count` are not part of tightness. In particular, `value_count` is the number of values across all rows written to the file and is not adjusted when rows are deleted; for a field with no repeated (`list` or `map`) ancestor, the live value count is `value_count` minus the deletion vector's cardinality. -When `tight_stats` is absent, readers must assume stats are not tight. A deletion vector does not by itself make a file's stats non-tight; `tight_stats` is the only indicator of tightness. Stats may be recomputed to restore tightness without rewriting the data file. +When `tight_stats` is absent, readers must assume stats are not tight; `tight_stats` is the only indicator of tightness. Stats may be recomputed to restore tightness without rewriting the data file. ###### Content Stats in Manifests From ad9d806f6bcfd7571760ee628503c007ecd0a31d Mon Sep 17 00:00:00 2001 From: Anoop Johnson Date: Tue, 20 Oct 2026 17:28:37 -0700 Subject: [PATCH 10/10] Spec: variant leaf bounds follow per-type rules --- format/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/format/spec.md b/format/spec.md index b27818c8076f..6d277e9a5007 100644 --- a/format/spec.md +++ b/format/spec.md @@ -1025,7 +1025,7 @@ Statistics are _tight_ when the bounds, `null_value_count`, and `nan_value_count Bounds for `string` and `binary` are never tight because they may be truncated when written. Bounds for `geometry` and `geography` are a bounding box, not exact minimum and maximum values. -`variant` bounds are stored per leaf field and are optional; a leaf field may have no bound, but any bound that is present is exact. A `variant` value is null only when the variant itself is null; a non-null variant that encodes a variant null is not counted as a null value. +`variant` bounds are stored per leaf field and are optional; a leaf field may have no bound, and each present bound follows the rules above for its type. A `variant` value is null only when the variant itself is null; a non-null variant that encodes a variant null is not counted as a null value. Statistics other than the bounds, `null_value_count`, and `nan_value_count` are not part of tightness. In particular, `value_count` is the number of values across all rows written to the file and is not adjusted when rows are deleted; for a field with no repeated (`list` or `map`) ancestor, the live value count is `value_count` minus the deletion vector's cardinality.