SPEC: Add File Type to v4 spec

format/spec.md

@@ -65,6 +65,7 @@ Version 4 of the Iceberg spec restructures metadata for improved performance and
 * Data files and their corresponding deletion vector are stored in a combined entry, removing the need for two-phase planning
 * Support for [relative locations](#file-locations-in-metadata) in metadata fields
 * Writing new [equality deletes](#equality-delete-files) is no longer allowed
+* New data type: `file`
 
 The full set of changes are listed in [Appendix E](#version-4).
 
@@ -235,7 +236,7 @@ When the `location` field is present in table metadata, it is used directly as t
 
 ### Schemas and Data Types
 
-A table's **schema** is a list of named columns. Data types are primitive, nested, or semi-structured. Nested types are maps, lists, or structs. A table schema is also a struct type.
+A table's **schema** is a list of named columns. Data types are primitive, nested, or semi-structured. Nested types are maps, lists, structs, or files. A table schema is also a struct type.
 
 For the representations of these types in Avro, ORC, and Parquet file formats, see Appendix A.
 
@@ -247,6 +248,8 @@ A **`list`** is a collection of values with some element type. The element field
 
 A **`map`** is a collection of key-value pairs with a key type and a value type. Both the key field and value field each have an integer id that is unique in the table schema. Map keys are required and map values can be either optional or required. Both map keys and map values may be any type, including nested types.
 
+A **`file`** stores a binary value, either inline or as a reference to an external file. The structure of this type is a fixed set of [nested fields](#file-type). These fields are referenced by offset from the field ID assigned to the encoding type, not represented directly in the schema.
+
 #### Semi-structured Types
 
 A **`variant`** is a value that stores semi-structured data. The structure and data types in a variant are not necessarily consistent across rows in a table or data file. The variant type and binary encoding are defined in the [Parquet project](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md), with support currently available for V1. Support for Variant is added in Iceberg v3.
@@ -328,6 +331,35 @@ For `geography` types, an additional parameter A specifies an algorithm for inte
 * `andoyer`: Thomas, Paul D. Mathematical models for navigation systems. US Naval Oceanographic Office, 1965.
 * `karney`: [Karney, Charles FF. "Algorithms for geodesics." Journal of Geodesy 87 (2013): 43-55](https://link.springer.com/content/pdf/10.1007/s00190-012-0578-z.pdf), and [GeographicLib](https://geographiclib.sourceforge.io/)
 
+#### File Type
+
+A **`file`** represents a range of bytes that may be stored inline as a value or as a reference to an external file. The `file` type and its value semantics are defined by the `FILE` logical type in the [Parquet project](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file).
+
+A `file` value has a fixed set of sub-fields. The sub-fields are implicit: they are not represented in the Iceberg schema and cannot be added, removed, reordered, or promoted. Their names, types, and field ID offsets are:
+
+| Sub-field      | ID offset | Type     |
+|----------------|-----------|----------|
+| `uri`          | 1         | `string` |
+| `offset`       | 2         | `long`   |
+| `size`         | 3         | `long`   |
+| `content_type` | 4         | `string` |
+| `checksum`     | 5         | `string` |
+| `inline`       | 6         | `binary` |
+
+A `file` field reserves the root field's ID plus six consecutive IDs for its sub-fields, assigned by the offsets above. Adding a `file` field must advance `last-column-id` to account for all 7 IDs.
+
+The `uri` field may contain absolute or relative references. Implementations that receive a relative path should resolve the path against the table location (see [Path Resolution](#path-resolution)).
+
+Statistics for `file` are tracked using separate field stats for each sub-field. Writers should produce statistics for `uri`, `content_type`, and `inline` fields; other fields may be omitted.
+
+A `file` column is subject to the following restrictions:
+
+* Equality, ordering, and hashing are not defined for `file` objects
+* A `file` column's `field-id` cannot be referenced as an identifier field or a source for partition or sort transforms.
+* A `file` is not interchangeable with a `struct`; a `struct` with the same sub-fields is not equivalent to a `file`.
+* A `file` column cannot be promoted from `struct` with the same sub-fields.
+* A `file` entry that references a `uri` does not guarantee its existence.
+
 #### Default values
 
 Default values can be tracked for struct fields (both nested structs and the top-level schema's struct). There can be two defaults with a field:
@@ -339,7 +371,7 @@ The `initial-default` is set only when a field is added to an existing schema. T
 
 The `initial-default` and `write-default` produce SQL default value behavior, without rewriting data files. SQL default value behavior when a field is added handles all existing rows as though the rows were written with the new field's default value. Default value changes may only affect future records and all known fields are written into data files. Omitting a known field when writing a data file is never allowed. The write default for a field must be written if a field is not supplied to a write. If the write default for a required field is not set, the writer must fail.
 
-All columns of `unknown`, `variant`, `geometry`, and `geography` types must default to null. Non-null values for `initial-default` or `write-default` are invalid.
+All columns of `unknown`, `variant`, `geometry`, `geography`, and `file` types must default to null. Non-null values for `initial-default` or `write-default` are invalid.
 
 Default values for the fields of a struct are tracked as `initial-default` and `write-default` at the field level. Default values for fields that are nested structs must not contain default values for the struct's fields (sub-fields). Sub-field defaults are tracked in sub-field's metadata. As a result, the default stored for a nested struct may be either null or a non-null struct with no field values. The effective default value is produced by setting each fields' default in a new struct.
 
@@ -1692,6 +1724,7 @@ Maps with non-string keys must use an array representation with the `map` logica
 |**`variant`**|`record` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. |Shredding is not supported in Avro.|
 |**`geometry`**|`bytes`|WKB format, see [Appendix G](#appendix-g-geospatial-notes)|
 |**`geography`**|`bytes`|WKB format, see [Appendix G](#appendix-g-geospatial-notes)|
+|**`file`**|`record` with the `file` sub-fields. Sub-fields must be assigned field IDs. |See [File Type](#file-type). Avro has no `FILE` logical type; type identity comes from the Iceberg schema.|
 
 Notes:
 
@@ -1747,6 +1780,7 @@ Lists must use the [3-level representation](https://github.com/apache/parquet-fo
 | **`variant`**      | `group` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs and the fields are accessed through names. | `VARIANT`                                   | See Parquet docs for [Variant encoding](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) and [Variant shredding encoding](https://github.com/apache/parquet-format/blob/master/VariantShredding.md). |
 | **`geometry`**     | `binary`                                                                                                                                     | `GEOMETRY`                                  | WKB format, see [Appendix G](#appendix-g-geospatial-notes).                             |
 | **`geography`**    | `binary`                                                                                                                                     | `GEOGRAPHY`                                 | WKB format, see [Appendix G](#appendix-g-geospatial-notes).                             |
+| **`file`**         | `group` with the `file` sub-fields. Sub-fields must be assigned field IDs.                                                                   | `FILE`                                      | See Parquet docs for the [`FILE` type](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#file) and [File Type](#file-type). |
 
 When reading an `unknown` column, any corresponding column must be ignored and replaced with `null` values.
 
@@ -1779,6 +1813,7 @@ When reading an `unknown` column, any corresponding column must be ignored and r
 | **`variant`**      | `struct` with `metadata` and `value` fields. `metadata` and `value` must not be assigned field IDs. |  `iceberg.struct-type`=`VARIANT`   | Shredding is not supported in ORC.                                                 |
 | **`geometry`**     | `binary`            | `iceberg.binary-type`=`GEOMETRY`                     | WKB format, see [Appendix G](#appendix-g-geospatial-notes).                                                      |
 | **`geography`**    | `binary`            | `iceberg.binary-type`=`GEOGRAPHY`                     | WKB format, see [Appendix G](#appendix-g-geospatial-notes).                                                      |
+| **`file`**         | `struct` with the `file` sub-fields. Sub-fields must be assigned and accessed by field IDs.| `iceberg.struct-type`=`FILE`| See [File Type](#file-type). |
 
 Notes:
 
@@ -1875,6 +1910,7 @@ Types are serialized according to this table:
 | **`variant`**| `JSON string: "variant"`|`"variant"`|
 | **`geometry(C)`** |`JSON string: "geometry(<C>)"`|`"geometry(srid:4326)"`|
 | **`geography(C, A)`** |`JSON string: "geography(<C>, <A>)"`|`"geography(srid:4326, spherical)"`|
+| **`file`**| `JSON string: "file"`|`"file"`|
 
 The schema JSON type strings in this table are the canonical serialized forms. Readers should accept optional whitespace around parameters and separators in parameterized type strings.
 
@@ -2034,6 +2070,7 @@ This serialization scheme is for storing single values as individual binary valu
 | **`variant`**                | Not supported                                                                                                |
 | **`geometry`**               | WKB format, see [Appendix G](#appendix-g-geospatial-notes)                                                   |
 | **`geography`**              | WKB format, see [Appendix G](#appendix-g-geospatial-notes)                                                   |
+| **`file`**                   | Not supported                                                                                                |
 
 ### Bound serialization
 
@@ -2077,6 +2114,8 @@ The binary single-value serialization can be used to store the lower and upper b
 
 ### Version 4
 
+The `file` type is added in v4.
+
 Relative path support is added in v4.
 
 Reading v3 or prior metadata for v4: