SPEC: Add File Type to v4 spec

[danielcweeks]

This PR adds language for the File type as defined in Parquet for V4.

[anoopj]

inline is the raw referenced bytes. Upper/lower bounds are potentially large and not useful for data skipping. The Delta RFC deliberately collects minValues/maxValues on uri, offset, size, content_type and explicitly excludes inline (and excludes checksum from min/max because a digest is uniformly distributed).

[danielcweeks]

The inline value is truncated per the bytes type. We felt it was useful to determine if there are inline or out-of-line values. You could omit the lower/upper bounds, but the value counts/nulls are meaningful.

Size and offset may be less important overall (offset is largely useless).

[dejankrak-db]

Aligned the Delta RFC delta-io/delta#7148 to include value counts/nulls for all fields.

[danielcweeks]

@stevenzwu can you clarify? This thread didn't say we were dropping those fields. I'm not sure I follow.

We're also not restricting or requiring specific stats or fields as they're all optional, so I don't follow your comment and it seems that further specification isn't necessary.

[stevenzwu]

sorry for the confusion.

The sub-fields are implicit: they are not represented in the Iceberg schema

if a sub field is not in the table schema, I thought it can't be stored in content_stats.

Adding a file field reserves the root field's ID plus six consecutive IDs for its sub-fields, assigned by the offsets above. Writers must advance last-column-id past all seven IDs and must not assign these IDs to any other field.

later the spec talked about last-column-id advancement rule. that is where I got a bit confused. Are we saying those are stats fields (with assigned field id) like Ryan's stats-only fields proposal? if that's the missing gap, it might be good to connect the dots in the spec text.

[danielcweeks]

All subfields have field ids and therefore can be represented in the content stats. There's no requirement that a field be represented in the schema to have stats; all that's required is to have a field-id so it can be represented.

[RussellSpitzer]

Added the minimal first step of implementation of this in #18252 which I believe matches this spec as written.

[Kurtiscwright]

@Kurtiscwright This can't be addressed at the type-definition level. Ownership and maintenance is a responsibility of the Catalog and who runts the maintenance procedure. The catalog should protect data access from secondary references trying to delete, but it's not a property of the type.

@danielcweeks I tried to stay away from the terminology Ownership because our community sync discussions made it clear that term invokes a lot of extra requirements outside the Specification scope of a data type.

I do think it's in Specification scope to have a well defined field, so it's cross catalog and cross engine, that can be used by a Maintenance operation to guide the correct decision on orphan clean up. I think this should be in the type definition because each entry in a table could be different, meaning in one row the file type could be something maintenance cleans up while the next is a file reference stored in multiple tables.

Separately I don't think it's a safe assumption that a table will mostly or only have files co-located/relative with a table. Especially with S3 Buckets being global.

[danielcweeks]

@Kurtiscwright This can't be addressed at the type-definition level. Ownership and maintenance is a responsibility of the Catalog and who runts the maintenance procedure. The catalog should protect data access from secondary references trying to delete, but it's not a property of the type.

@danielcweeks I tried to stay away from the terminology Ownership because our community sync discussions made it clear that term invokes a lot of extra requirements outside the Specification scope of a data type.

I do think it's in Specification scope to have a well defined field, so it's cross catalog and cross engine, that can be used by a Maintenance operation to guide the correct decision on orphan clean up. I think this should be in the type definition because each entry in a table could be different, meaning in one row the file type could be something maintenance cleans up while the next is a file reference stored in multiple tables.

Separately I don't think it's a safe assumption that a table will mostly or only have files co-located/relative with a table. Especially with S3 Buckets being global.

@Kurtiscwright I think we should make this a separate discussion from the spec definition of the type. This is consistent with all other files that may be shared (manifests, data files, etc. even if not recommended). We don't define who "owns" these artifacts as part of the spec and adding a value to the type makes this more complicated (not less). You still have all of the ownership issues if you add it to the type, but now you have an additional field that can be out of sync (ownership transfer, replication, etc. can all result in inconsistent semantics).

We're also not making assumptions about co-location/relative files, but rather, defining the representations and requirements for absolute and relative pathing. I think there will be use cases where all references are external (e.g. to a media catalog) and there will be cases where they will all be relative (e.g. ingest pipelines), so we want well defined expectations without breaking other gauarantees.