Spec: Replace per-column tight_bounds with file-level tight_stats in v4 by anoopj · Pull Request #18585 · apache/iceberg

anoopj · 2026-10-19T18:22:25Z

Track stats tightness with a single file-level tight_stats flag on the tracked file instead of a per-column tight_bounds field. The flag covers all statistics (bounds and null, NaN, and value counts) for the file's live rows.

Track stats tightness with a single file-level tight_stats flag on the tracked file instead of a per-column tight_bounds field. The flag covers all statistics (bounds and null, NaN, and value counts) for the file's live rows.

dramaticlly · 2026-10-19T18:58:17Z

+When `tight_stats` is true:
+
+* Counts are exact for all fields.
+* Bounds are exact for fixed-width primitives and for `variant`.


do we consider the variant bounds to be exact?

Good callout. I think variant is a bit different in the sense that some of the fields may not have stats, but any field level stats present has the same tightness guarantees. I will try to incorporate this.

anoopj · 2026-10-19T23:23:43Z

+* Bounds are the minimum and maximum of the live values.
+* `null_value_count` and `nan_value_count` are the counts across the live rows.
+
+`value_count` and `total_bytes` are not part of tightness. `value_count` is the number of values across all rows written to the file and is not adjusted when rows are deleted; for a field with no repeated (`list` or `map`) ancestor, the live value count is `value_count` minus the deletion vector's cardinality. `total_bytes` is an estimate and is likewise not adjusted for deleted rows.


Verified against the implementation that computes the value_count:

value_count counts every row, including rows where a parent struct is null

Parquet's per-column value count is one entry per row regardless of definition level, so a row whose parent struct is null still contributes to the leaf's value_count.

Parquet marks a leaf null whenever its definition level is below max, which includes the case where an ancestor struct is null, so a null parent also increments the leaf's null_value_count.

Repeated (list/map) fields collect no stats at all

So the derivation logic above to calculate value_count by subtracting dv cardinality is correct.

Not spelled out here, but similarly non-null values can be calculated by value_count - dv_cardinality - null_count (assuming no repeated ancestors).

rdblue · 2026-10-19T23:59:21Z

        - `column_files` may only be set if the tracked file is a data file or a v4 leaf manifest.
        - A field ID may appear in `field_ids` of at most one column file in a tracked file's `column_files`.
+    - `tight_stats`:
+        - `tight_stats` may be true only if `content_stats` are tight; attaching or updating a `deletion_vector` without recomputing stats makes them no longer tight. See [Stats Tightness](#stats-tightness).


Do we need to mention a DV here? You don't necessarily need to recompute stats and getting into the nuance should be done in more detail below. I'd probably just remove that sentence.

rdblue · 2026-10-19T23:59:51Z


 For `variant`, both bounds are unshredded `variant` that store variant field bounds by normalized JSON paths as field names. See [Bounds for Variant](#bounds-for-variant) for details on producing these bounds.

+<a id="stats-tightness"></a>


Isn't this automatically generated by the heading on the next line?

yeah we shouldn't need to generate this

Looks like the anchor tag will be autogenerated for headings. I have removed it.

rdblue · 2026-10-20T00:00:20Z

+<a id="stats-tightness"></a>
+###### Stats Tightness
+
+Statistics are *tight* when they are exact for the live rows of a file:


When bounds, null count, and nan count are ...

Otherwise this is confusing for the other stats fields.

Done. The definition now names them (bounds, null and NaN counts). The rest are clearly out.

rdblue · 2026-10-20T00:00:47Z

+* Bounds are the minimum and maximum of the live values.
+* `null_value_count` and `nan_value_count` are the counts across the live rows.
+
+`value_count` and `total_bytes` are not part of tightness. `value_count` is the number of values across all rows written to the file and is not adjusted when rows are deleted; for a field with no repeated (`list` or `map`) ancestor, the live value count is `value_count` minus the deletion vector's cardinality. `total_bytes` is an estimate and is likewise not adjusted for deleted rows.


"Note that value_count is not included in tightness"

We don't need to state that total bytes is excluded. We should take care of that with the definition of tightness that only mentions null and nan counts in addition to the bounds. All other fields are not tight and we can't require new ones to be included either.

Done. Dropped the total_bytes sentence and reframed it as:

Statistics other than the bounds, null_value_count, and nan_value_count are not part of tightness.

rdblue · 2026-10-20T00:02:46Z

+
+When `tight_stats` is true:
+
+* `null_value_count` and `nan_value_count` are exact for the live rows.


This is restating the definition above and should be removed.

rdblue · 2026-10-20T00:03:38Z

+When `tight_stats` is true:
+
+* `null_value_count` and `nan_value_count` are exact for the live rows.
+* Bounds are exact for primitive types other than `string` and `binary`, whose bounds may have been truncated when written.


I think this should also be part of the definition above. Bounds for string and binary fields are excluded and are never tight.

Moved into the definition. I folded geo in there too, since it's the same kind of exception. Keeping it under the tight_stats was misleading.

rdblue · 2026-10-20T00:04:19Z

+* `variant` bounds are stored per leaf field and are optional; a leaf field may have no bound, but any bound that is present is exact.
+* `geometry` and `geography` bounds are a bounding box that contains the live values, not exact minimum and maximum values.
+
+A `variant` value is null only when the variant itself is null; a non-null variant that encodes a variant null is not counted as a null value. Variants do not have a `nan_value_count`.


This is good. We don't need to specify that variants don't have a NaN count though.

Thanks. I dropped the NaN-count line, kept the variant-null clause.

rdblue · 2026-10-20T18:09:32Z

        - `column_files` may only be set if the tracked file is a data file or a v4 leaf manifest.
        - A field ID may appear in `field_ids` of at most one column file in a tracked file's `column_files`.
+    - `tight_stats`:
+        - `tight_stats` may be true only if `content_stats` are tight. See [Stats Tightness](#stats-tightness).


This restates the description. I would remove it.

rdblue · 2026-10-20T18:12:08Z


+###### Stats Tightness
+
+Statistics are *tight* when the bounds, `null_value_count`, and `nan_value_count` are exact for the live rows of a file:


Is the intent to use bold or italic? Most other places use _underscores_ for italic or **double asterisks** for bold.

rdblue · 2026-10-20T18:14:59Z

+
+Statistics are *tight* when the bounds, `null_value_count`, and `nan_value_count` are exact for the live rows of a file:
+
+* Bounds are the minimum and maximum of the live values. Bounds for `string` and `binary` are never tight because they may be truncated when written, and bounds for `geometry` and `geography` are a bounding box, not exact minimum and maximum values.


lower_bound and upper_bound for fixed-width types are equal to the minimum and maximum live values

The second sentence should not be in the definition, it is a clarification that can come later. The idea is to be clear and direct and "fixed-width types" says what we need to. Leaving aside the specific cases makes this easier to understand.

rdblue · 2026-10-20T18:16:30Z

+
+* Bounds are the minimum and maximum of the live values. Bounds for `string` and `binary` are never tight because they may be truncated when written, and bounds for `geometry` and `geography` are a bounding box, not exact minimum and maximum values.
+* `variant` bounds are stored per leaf field and are optional; a leaf field may have no bound, but any bound that is present is exact.
+* `null_value_count` and `nan_value_count` are the counts across the live rows. A `variant` value is null only when the variant itself is null; a non-null variant that encodes a variant null is not counted as a null value.


This should be the second bullet point.

Information about variant can be its own paragraph below. We don't need to bring exceptions in when defining what it means for bounds to be exact.

rdblue · 2026-10-20T18:19:46Z

+Statistics are *tight* when the bounds, `null_value_count`, and `nan_value_count` are exact for the live rows of a file:
+
+* Bounds are the minimum and maximum of the live values. Bounds for `string` and `binary` are never tight because they may be truncated when written, and bounds for `geometry` and `geography` are a bounding box, not exact minimum and maximum values.
+* `variant` bounds are stored per leaf field and are optional; a leaf field may have no bound, but any bound that is present is exact.


Also move this to a separate paragraph about variants.

github-actions Bot added the Specification Issues that may introduce spec changes. label Oct 19, 2026

dramaticlly reviewed Oct 19, 2026

View reviewed changes

anoopj added 5 commits October 19, 2026 13:16

Spec: Clarify tight_stats bound exactness for variant and geo types

44294ca

clarify variant and geometry

5d57963

Spec: Refine v4 stats tightness for counts

448217d

Drop the part about live rows

61f0eb4

Fix total_bytes wording

b9675a7

anoopj commented Oct 19, 2026

View reviewed changes

rdblue reviewed Oct 19, 2026

View reviewed changes

rdblue reviewed Oct 20, 2026

View reviewed changes

Spec: Address review feedback on stats tightness

1bbf989

anoopj requested review from dramaticlly, nastra and rdblue October 20, 2026 17:39

rdblue reviewed Oct 20, 2026

View reviewed changes


		For `variant`, both bounds are unshredded `variant` that store variant field bounds by normalized JSON paths as field names. See [Bounds for Variant](#bounds-for-variant) for details on producing these bounds.

		<a id="stats-tightness"></a>


		When `tight_stats` is true:

		* `null_value_count` and `nan_value_count` are exact for the live rows.


		###### Stats Tightness

		Statistics are tight when the bounds, `null_value_count`, and `nan_value_count` are exact for the live rows of a file:


		Statistics are tight when the bounds, `null_value_count`, and `nan_value_count` are exact for the live rows of a file:

		* Bounds are the minimum and maximum of the live values. Bounds for `string` and `binary` are never tight because they may be truncated when written, and bounds for `geometry` and `geography` are a bounding box, not exact minimum and maximum values.

Conversation

anoopj commented Oct 19, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblue Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rdblue Oct 20, 2026 •

edited

Loading