Add Reference Tables

dev/.rat-excludes

@@ -26,3 +26,7 @@ node_modules/**
 # Apache header; they are covered by the repository LICENSE and NOTICE at the root.
 **/*.json
 **/*.jsonl
+
+# Avro container files embed their schema as JSON text in the file header, so RAT
+# reads them as text. They are binary fixture data and cannot carry a header either.
+**/*.avro

reference-tables/README.md

@@ -0,0 +1,186 @@
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+# Reference tables
+
+This directory holds complete Iceberg tables: metadata, manifest lists,
+manifests, and Parquet data files. Tables are grouped by format version.
+
+## Assertion
+
+```
+scan(table, snapshot) == rows
+```
+
+For every snapshot a table's metadata lists, a full scan at that snapshot returns
+exactly the rows in the table's `expected/scan.json`. Bytes are not compared.
+
+## Size
+
+A table should stay under 100 KB. The four here are 16 KB to 29 KB.
+
+Size tracks commit count, and almost nothing else. A commit costs roughly 12 to
+14 KB, over half of it the manifest and manifest list it produces. Rows are close
+to free: a Parquet file holding five rows is under 1 KB and is nearly all footer.
+
+So the limit is about seven commits. A table that needs more should say why in
+its pull request.
+
+## Paths
+
+Prior to v4 the spec requires every path field to be fully qualified, so each
+table carries an absolute prefix in `location`, in every `manifest-list`, and in
+every `file_path`. That prefix is the sentinel root:
+
+```
+file:///tmp/iceberg-verification/reference-tables/<format-version>/<table-name>
+```
+
+A consumer resolves the prefix by symlinking `/tmp/iceberg-verification` to the
+root of its checkout, which leaves every committed byte untouched:
+
+```
+ln -s "$(pwd)" /tmp/iceberg-verification
+```
+
+This route is verified on Linux. It is not yet verified on Windows.
+
+Starting with v4 the spec permits relative paths and makes `location` optional,
+so a v4 table needs no symlink.
+
+Loading a table opens only its metadata file, so a load succeeds without the
+symlink. The scan in the assertion opens every manifest list, manifest and data
+file, so a passing scan also shows the symlink resolves.
+
+## What is here
+
+`index.json` lists every table: its name, format version, the implementation and
+release that wrote it, the path to its metadata file, and a one-line description.
+Consumers read it; the `description` field is documentation and is ignored.
+
+```json
+{
+  "tables": [
+    {
+      "name": "v2/simple-append-parquet-data",
+      "format-version": 2,
+      "written-by": "Iceberg Java 1.11.0",
+      "data-file-format": "parquet",
+      "metadata": "v2/simple-append-parquet-data/metadata/v3.metadata.json",
+      "description": "Flat schema of long, string and timestamptz. Two appends, four rows, one null."
+    }
+  ]
+}
+```
+
+A table is written once and not regenerated, so `written-by` is per table: the
+corpus holds whatever version was current when each table was added.
+
+## Naming
+
+A table directory names every choice an implementation has to support in order
+to run it. Today that is the format version, which is the parent directory, and
+the data file format, which is the suffix:
+
+```
+reference-tables/<format-version>/<what-it-covers>-<data-file-format>-data
+```
+
+So `v2/simple-append-parquet-data` is readable without opening anything: a v2
+table whose data files are Parquet. The suffix names the data files specifically,
+because the manifests in the same directory are Avro and the metadata is JSON. An
+implementation that reads no Parquet skips it by path, without parsing the index.
+The same choices appear as fields in `index.json` for consumers that would rather
+filter there.
+
+A later fixture that varies something else, delete file format for instance,
+carries that in the name too.
+
+## Entry point
+
+Resolve a table's `metadata` path from `index.json` and open it as a static
+table. Every implementation takes an explicit metadata file path:
+
+| | |
+| -- | -- |
+| Iceberg Java | `new BaseTable(new StaticTableOperations(path, io), name)` |
+| PyIceberg | `StaticTable.from_metadata(path)` |
+| iceberg-rust | `StaticTable::from_metadata_file(path, ident, io)` |
+| iceberg-go | `table.NewFromLocation(..., path, ...)` |
+| iceberg-cpp | `TableMetadataUtil::Read(io, path)`, then `StaticTable::Make(ident, metadata, path, io)` |
+
+No catalog is involved and no directory listing is needed.
+
+The corpus carries no `metadata/version-hint.text`. That is a Hadoop convention
+rather than a spec artifact, and only Iceberg Java and PyIceberg read it. An
+explicit metadata path is the one mechanism all five share.
+
+Only the current `metadata.json` is committed. Its name carries a version
+counter, not the table's format version: a writer names each metadata commit
+`v<V>.metadata.json` starting at `v1` on create, so `v1/partitioned-parquet-data`
+is a format v1 table whose metadata file is `v3.metadata.json` after two appends.
+The directory name is the format version. The file name is not. Read the path
+from the index rather than constructing it.
+
+Earlier metadata files are pruned to reduce clutter. `metadata-log` still lists
+them, which is correct: it is a record of past commits, not a set of live
+references, and nothing resolves those paths on load.
+
+## Expected values
+
+Each table directory holds `expected/scan.json`, with one entry per snapshot:
+
+```json
+{
+  "snapshots": [
+    {
+      "snapshot-id": "7973649908883454602",
+      "rows": [
+        {"1": "1", "2": "alpha", "3": "2026-01-01T00:00:00.000000+00:00"},
+        {"1": "2", "2": "bravo", "3": "2026-01-02T00:00:00.000000+00:00"}
+      ]
+    }
+  ]
+}
+```
+
+- `snapshot-id` is the snapshot scanned.
+- `rows` is every row the scan returns at that snapshot. Each row is a struct of
+  the table schema, written in the spec's
+  [JSON single-value serialization](https://github.com/apache/iceberg/blob/main/format/spec.md#json-single-value-serialization):
+  keyed by field ID, with every nested value in the same form.
+
+`snapshot-id` and every `long` value are written as strings, following the
+64-bit integer rule in `CONTRIBUTING.md`. This is the one departure from the
+single-value serialization, which writes a `long` as a JSON number.
+
+## Procedure
+
+Each entry in `snapshots` is checked on its own. The table is scanned at that
+snapshot through the implementation's scan API, with no filter and every column
+of the schema projected. Reading the data files directly bypasses scan planning,
+so it does not satisfy the assertion.
+
+Each value is decoded as the type the table schema gives its field ID, and
+compared as that decoded value. The returned rows and `rows` are then compared
+as multisets, because the spec fixes no row order for a scan. List elements keep
+their order. Map entries are compared as a set of key and value pairs, because
+a map fixes no entry order.
+
+Every difference is reported, naming the table, the snapshot, and the row.

reference-tables/index.json

@@ -0,0 +1,36 @@
+{
+  "tables": [
+    {
+      "name": "v2/simple-append-parquet-data",
+      "format-version": 2,
+      "written-by": "Iceberg Java 1.11.0",
+      "data-file-format": "parquet",
+      "metadata": "v2/simple-append-parquet-data/metadata/v3.metadata.json",
+      "description": "Flat schema of long, string and timestamptz. Two appends, four rows, one null."
+    },
+    {
+      "name": "v2/nested-types-parquet-data",
+      "format-version": 2,
+      "written-by": "Iceberg Java 1.11.0",
+      "data-file-format": "parquet",
+      "metadata": "v2/nested-types-parquet-data/metadata/v2.metadata.json",
+      "description": "Struct, list and map, so every nested id space appears. One snapshot, four rows: populated containers, null containers, empty containers, and a null list element and null map value."
+    },
+    {
+      "name": "v1/partitioned-parquet-data",
+      "format-version": 1,
+      "written-by": "Iceberg Java 1.11.0",
+      "data-file-format": "parquet",
+      "metadata": "v1/partitioned-parquet-data/metadata/v3.metadata.json",
+      "description": "Identity partition on category and a day transform on event_ts. Two snapshots, one data file per partition, three rows."
+    },
+    {
+      "name": "v2/partitioned-parquet-data",
+      "format-version": 2,
+      "written-by": "Iceberg Java 1.11.0",
+      "data-file-format": "parquet",
+      "metadata": "v2/partitioned-parquet-data/metadata/v3.metadata.json",
+      "description": "Identity partition on category and a day transform on event_ts. Two snapshots, one data file per partition, three rows."
+    }
+  ]
+}

reference-tables/v1/partitioned-parquet-data/expected/scan.json

@@ -0,0 +1,19 @@
+{
+  "snapshots": [
+    {
+      "snapshot-id": "1602692989443393359",
+      "rows": [
+        {"1": "1", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"},
+        {"1": "2", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"}
+      ]
+    },
+    {
+      "snapshot-id": "5699608042525680694",
+      "rows": [
+        {"1": "1", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"},
+        {"1": "2", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"},
+        {"1": "3", "2": "music", "3": "2026-01-02T00:00:00.000000+00:00"}
+      ]
+    }
+  ]
+}

reference-tables/v1/partitioned-parquet-data/metadata/v3.metadata.json

@@ -0,0 +1 @@
+{"format-version":1,"table-uuid":"a79c6b9f-76a3-472e-a4ed-85ff532ccd11","location":"file:///tmp/iceberg-verification/reference-tables/v1/partitioned-parquet-data","last-updated-ms":1788557522647,"last-column-id":3,"schema":{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":true,"type":"long"},{"id":2,"name":"category","required":true,"type":"string"},{"id":3,"name":"event_ts","required":true,"type":"timestamptz"}]},"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":true,"type":"long"},{"id":2,"name":"category","required":true,"type":"string"},{"id":3,"name":"event_ts","required":true,"type":"timestamptz"}]}],"partition-spec":[{"name":"category","transform":"identity","source-id":2,"field-id":1000},{"name":"event_day","transform":"day","source-id":3,"field-id":1001}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"category","transform":"identity","source-id":2,"field-id":1000},{"name":"event_day","transform":"day","source-id":3,"field-id":1001}]}],"last-partition-id":1001,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"write.parquet.compression-codec":"zstd"},"current-snapshot-id":5699608042525680694,"refs":{"main":{"snapshot-id":5699608042525680694,"type":"branch"}},"snapshots":[{"snapshot-id":1602692989443393359,"timestamp-ms":1788557522539,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"0","manifests-replaced":"0","added-data-files":"1","added-records":"2","added-files-size":"1087","changed-partition-count":"1","total-records":"2","total-files-size":"1087","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v1/partitioned-parquet-data/metadata/snap-1602692989443393359-1-b0361de6-42e9-4841-ad9c-87a60d52d11d.avro","schema-id":0},{"snapshot-id":5699608042525680694,"parent-snapshot-id":1602692989443393359,"timestamp-ms":1788557522647,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"1","manifests-replaced":"0","added-data-files":"1","added-records":"1","added-files-size":"979","changed-partition-count":"1","total-records":"3","total-files-size":"2066","total-data-files":"2","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v1/partitioned-parquet-data/metadata/snap-5699608042525680694-1-516446c4-d70f-421a-af9a-50fe8654ebbd.avro","schema-id":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1788557522539,"snapshot-id":1602692989443393359},{"timestamp-ms":1788557522647,"snapshot-id":5699608042525680694}],"metadata-log":[{"timestamp-ms":1788557522406,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v1/partitioned-parquet-data/metadata/v1.metadata.json"},{"timestamp-ms":1788557522539,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v1/partitioned-parquet-data/metadata/v2.metadata.json"}]}

reference-tables/v2/nested-types-parquet-data/expected/scan.json

@@ -0,0 +1,13 @@
+{
+  "snapshots": [
+    {
+      "snapshot-id": "9153710085634724558",
+      "rows": [
+        {"1": "1", "2": {"5": 40.7128, "6": -74.006}, "3": ["east", "coastal"], "4": {"keys": ["pop"], "values": [8336817]}},
+        {"1": "2", "2": null, "3": null, "4": null},
+        {"1": "3", "2": {"5": 51.5074, "6": -0.1278}, "3": [], "4": {"keys": [], "values": []}},
+        {"1": "4", "2": null, "3": ["west", null], "4": {"keys": ["pop"], "values": [null]}}
+      ]
+    }
+  ]
+}

reference-tables/v2/nested-types-parquet-data/metadata/v2.metadata.json

@@ -0,0 +1 @@
+{"format-version":2,"table-uuid":"be762ba3-cef6-4d04-81b4-033c7db9b38e","location":"file:///tmp/iceberg-verification/reference-tables/v2/nested-types-parquet-data","last-sequence-number":1,"last-updated-ms":1792616362118,"last-column-id":9,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":true,"type":"long"},{"id":2,"name":"location","required":false,"type":{"type":"struct","fields":[{"id":5,"name":"lat","required":true,"type":"double"},{"id":6,"name":"lon","required":true,"type":"double"}]}},{"id":3,"name":"tags","required":false,"type":{"type":"list","element-id":7,"element":"string","element-required":false}},{"id":4,"name":"props","required":false,"type":{"type":"map","key-id":8,"key":"string","value-id":9,"value":"int","value-required":false}}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"last-partition-id":999,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"write.parquet.compression-codec":"zstd"},"current-snapshot-id":9153710085634724558,"refs":{"main":{"snapshot-id":9153710085634724558,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":9153710085634724558,"timestamp-ms":1792616362084,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"0","manifests-replaced":"0","added-data-files":"1","added-records":"4","added-files-size":"2167","changed-partition-count":"1","total-records":"4","total-files-size":"2167","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v2/nested-types-parquet-data/metadata/snap-9153710085634724558-1-35668469-d7a9-4d61-9da5-c65b78dd5407.avro","schema-id":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1792616362084,"snapshot-id":9153710085634724558}],"metadata-log":[{"timestamp-ms":1792616360584,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v2/nested-types-parquet-data/metadata/v1.metadata.json"}]}

reference-tables/v2/partitioned-parquet-data/expected/scan.json

@@ -0,0 +1,19 @@
+{
+  "snapshots": [
+    {
+      "snapshot-id": "464121021900750417",
+      "rows": [
+        {"1": "1", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"},
+        {"1": "2", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"}
+      ]
+    },
+    {
+      "snapshot-id": "8206603330787222691",
+      "rows": [
+        {"1": "1", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"},
+        {"1": "2", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"},
+        {"1": "3", "2": "music", "3": "2026-01-02T00:00:00.000000+00:00"}
+      ]
+    }
+  ]
+}

reference-tables/v2/partitioned-parquet-data/metadata/v3.metadata.json

@@ -0,0 +1 @@
+{"format-version":2,"table-uuid":"89a49230-98bc-4b4f-b62c-9aa60d32713b","location":"file:///tmp/iceberg-verification/reference-tables/v2/partitioned-parquet-data","last-sequence-number":2,"last-updated-ms":1790699028599,"last-column-id":3,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":true,"type":"long"},{"id":2,"name":"category","required":true,"type":"string"},{"id":3,"name":"event_ts","required":true,"type":"timestamptz"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"category","transform":"identity","source-id":2,"field-id":1000},{"name":"event_day","transform":"day","source-id":3,"field-id":1001}]}],"last-partition-id":1001,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"write.parquet.compression-codec":"zstd"},"current-snapshot-id":8206603330787222691,"refs":{"main":{"snapshot-id":8206603330787222691,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":464121021900750417,"timestamp-ms":1790699028377,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"0","manifests-replaced":"0","added-data-files":"1","added-records":"2","added-files-size":"1087","changed-partition-count":"1","total-records":"2","total-files-size":"1087","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v2/partitioned-parquet-data/metadata/snap-464121021900750417-1-a0a27bf6-f023-4840-838f-5513264f8deb.avro","schema-id":0},{"sequence-number":2,"snapshot-id":8206603330787222691,"parent-snapshot-id":464121021900750417,"timestamp-ms":1790699028599,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"1","manifests-replaced":"0","added-data-files":"1","added-records":"1","added-files-size":"979","changed-partition-count":"1","total-records":"3","total-files-size":"2066","total-data-files":"2","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v2/partitioned-parquet-data/metadata/snap-8206603330787222691-1-b82906e8-8a4d-496c-8461-6420f7764946.avro","schema-id":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1790699028377,"snapshot-id":464121021900750417},{"timestamp-ms":1790699028599,"snapshot-id":8206603330787222691}],"metadata-log":[{"timestamp-ms":1790699027227,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v2/partitioned-parquet-data/metadata/v1.metadata.json"},{"timestamp-ms":1790699028392,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v2/partitioned-parquet-data/metadata/v2.metadata.json"}]}

reference-tables/v2/simple-append-parquet-data/expected/scan.json

@@ -0,0 +1,20 @@
+{
+  "snapshots": [
+    {
+      "snapshot-id": "7973649908883454602",
+      "rows": [
+        {"1": "1", "2": "alpha", "3": "2026-01-01T00:00:00.000000+00:00"},
+        {"1": "2", "2": "bravo", "3": "2026-01-02T00:00:00.000000+00:00"}
+      ]
+    },
+    {
+      "snapshot-id": "8192414647100221600",
+      "rows": [
+        {"1": "1", "2": "alpha", "3": "2026-01-01T00:00:00.000000+00:00"},
+        {"1": "2", "2": "bravo", "3": "2026-01-02T00:00:00.000000+00:00"},
+        {"1": "3", "2": "charlie", "3": "2026-01-03T00:00:00.000000+00:00"},
+        {"1": "4", "2": null, "3": "2026-01-04T00:00:00.000000+00:00"}
+      ]
+    }
+  ]
+}