diff --git a/dev/.rat-excludes b/dev/.rat-excludes index 5830bfb..723fd3d 100644 --- a/dev/.rat-excludes +++ b/dev/.rat-excludes @@ -26,3 +26,7 @@ node_modules/** # Apache header; they are covered by the repository LICENSE and NOTICE at the root. **/*.json **/*.jsonl + +# Avro container files embed their schema as JSON text in the file header, so RAT +# reads them as text. They are binary fixture data and cannot carry a header either. +**/*.avro diff --git a/reference-tables/README.md b/reference-tables/README.md new file mode 100644 index 0000000..6905346 --- /dev/null +++ b/reference-tables/README.md @@ -0,0 +1,186 @@ + + +# Reference tables + +This directory holds complete Iceberg tables: metadata, manifest lists, +manifests, and Parquet data files. Tables are grouped by format version. + +## Assertion + +``` +scan(table, snapshot) == rows +``` + +For every snapshot a table's metadata lists, a full scan at that snapshot returns +exactly the rows in the table's `expected/scan.json`. Bytes are not compared. + +## Size + +A table should stay under 100 KB. The four here are 16 KB to 29 KB. + +Size tracks commit count, and almost nothing else. A commit costs roughly 12 to +14 KB, over half of it the manifest and manifest list it produces. Rows are close +to free: a Parquet file holding five rows is under 1 KB and is nearly all footer. + +So the limit is about seven commits. A table that needs more should say why in +its pull request. + +## Paths + +Prior to v4 the spec requires every path field to be fully qualified, so each +table carries an absolute prefix in `location`, in every `manifest-list`, and in +every `file_path`. That prefix is the sentinel root: + +``` +file:///tmp/iceberg-verification/reference-tables// +``` + +A consumer resolves the prefix by symlinking `/tmp/iceberg-verification` to the +root of its checkout, which leaves every committed byte untouched: + +``` +ln -s "$(pwd)" /tmp/iceberg-verification +``` + +This route is verified on Linux. It is not yet verified on Windows. + +Starting with v4 the spec permits relative paths and makes `location` optional, +so a v4 table needs no symlink. + +Loading a table opens only its metadata file, so a load succeeds without the +symlink. The scan in the assertion opens every manifest list, manifest and data +file, so a passing scan also shows the symlink resolves. + +## What is here + +`index.json` lists every table: its name, format version, the implementation and +release that wrote it, the path to its metadata file, and a one-line description. +Consumers read it; the `description` field is documentation and is ignored. + +```json +{ + "tables": [ + { + "name": "v2/simple-append-parquet-data", + "format-version": 2, + "written-by": "Iceberg Java 1.11.0", + "data-file-format": "parquet", + "metadata": "v2/simple-append-parquet-data/metadata/v3.metadata.json", + "description": "Flat schema of long, string and timestamptz. Two appends, four rows, one null." + } + ] +} +``` + +A table is written once and not regenerated, so `written-by` is per table: the +corpus holds whatever version was current when each table was added. + +## Naming + +A table directory names every choice an implementation has to support in order +to run it. Today that is the format version, which is the parent directory, and +the data file format, which is the suffix: + +``` +reference-tables//--data +``` + +So `v2/simple-append-parquet-data` is readable without opening anything: a v2 +table whose data files are Parquet. The suffix names the data files specifically, +because the manifests in the same directory are Avro and the metadata is JSON. An +implementation that reads no Parquet skips it by path, without parsing the index. +The same choices appear as fields in `index.json` for consumers that would rather +filter there. + +A later fixture that varies something else, delete file format for instance, +carries that in the name too. + +## Entry point + +Resolve a table's `metadata` path from `index.json` and open it as a static +table. Every implementation takes an explicit metadata file path: + +| | | +| -- | -- | +| Iceberg Java | `new BaseTable(new StaticTableOperations(path, io), name)` | +| PyIceberg | `StaticTable.from_metadata(path)` | +| iceberg-rust | `StaticTable::from_metadata_file(path, ident, io)` | +| iceberg-go | `table.NewFromLocation(..., path, ...)` | +| iceberg-cpp | `TableMetadataUtil::Read(io, path)`, then `StaticTable::Make(ident, metadata, path, io)` | + +No catalog is involved and no directory listing is needed. + +The corpus carries no `metadata/version-hint.text`. That is a Hadoop convention +rather than a spec artifact, and only Iceberg Java and PyIceberg read it. An +explicit metadata path is the one mechanism all five share. + +Only the current `metadata.json` is committed. Its name carries a version +counter, not the table's format version: a writer names each metadata commit +`v.metadata.json` starting at `v1` on create, so `v1/partitioned-parquet-data` +is a format v1 table whose metadata file is `v3.metadata.json` after two appends. +The directory name is the format version. The file name is not. Read the path +from the index rather than constructing it. + +Earlier metadata files are pruned to reduce clutter. `metadata-log` still lists +them, which is correct: it is a record of past commits, not a set of live +references, and nothing resolves those paths on load. + +## Expected values + +Each table directory holds `expected/scan.json`, with one entry per snapshot: + +```json +{ + "snapshots": [ + { + "snapshot-id": "7973649908883454602", + "rows": [ + {"1": "1", "2": "alpha", "3": "2026-01-01T00:00:00.000000+00:00"}, + {"1": "2", "2": "bravo", "3": "2026-01-02T00:00:00.000000+00:00"} + ] + } + ] +} +``` + +- `snapshot-id` is the snapshot scanned. +- `rows` is every row the scan returns at that snapshot. Each row is a struct of + the table schema, written in the spec's + [JSON single-value serialization](https://github.com/apache/iceberg/blob/main/format/spec.md#json-single-value-serialization): + keyed by field ID, with every nested value in the same form. + +`snapshot-id` and every `long` value are written as strings, following the +64-bit integer rule in `CONTRIBUTING.md`. This is the one departure from the +single-value serialization, which writes a `long` as a JSON number. + +## Procedure + +Each entry in `snapshots` is checked on its own. The table is scanned at that +snapshot through the implementation's scan API, with no filter and every column +of the schema projected. Reading the data files directly bypasses scan planning, +so it does not satisfy the assertion. + +Each value is decoded as the type the table schema gives its field ID, and +compared as that decoded value. The returned rows and `rows` are then compared +as multisets, because the spec fixes no row order for a scan. List elements keep +their order. Map entries are compared as a set of key and value pairs, because +a map fixes no entry order. + +Every difference is reported, naming the table, the snapshot, and the row. diff --git a/reference-tables/index.json b/reference-tables/index.json new file mode 100644 index 0000000..c41fb8c --- /dev/null +++ b/reference-tables/index.json @@ -0,0 +1,36 @@ +{ + "tables": [ + { + "name": "v2/simple-append-parquet-data", + "format-version": 2, + "written-by": "Iceberg Java 1.11.0", + "data-file-format": "parquet", + "metadata": "v2/simple-append-parquet-data/metadata/v3.metadata.json", + "description": "Flat schema of long, string and timestamptz. Two appends, four rows, one null." + }, + { + "name": "v2/nested-types-parquet-data", + "format-version": 2, + "written-by": "Iceberg Java 1.11.0", + "data-file-format": "parquet", + "metadata": "v2/nested-types-parquet-data/metadata/v2.metadata.json", + "description": "Struct, list and map, so every nested id space appears. One snapshot, four rows: populated containers, null containers, empty containers, and a null list element and null map value." + }, + { + "name": "v1/partitioned-parquet-data", + "format-version": 1, + "written-by": "Iceberg Java 1.11.0", + "data-file-format": "parquet", + "metadata": "v1/partitioned-parquet-data/metadata/v3.metadata.json", + "description": "Identity partition on category and a day transform on event_ts. Two snapshots, one data file per partition, three rows." + }, + { + "name": "v2/partitioned-parquet-data", + "format-version": 2, + "written-by": "Iceberg Java 1.11.0", + "data-file-format": "parquet", + "metadata": "v2/partitioned-parquet-data/metadata/v3.metadata.json", + "description": "Identity partition on category and a day transform on event_ts. Two snapshots, one data file per partition, three rows." + } + ] +} diff --git a/reference-tables/v1/partitioned-parquet-data/data/a.parquet b/reference-tables/v1/partitioned-parquet-data/data/a.parquet new file mode 100644 index 0000000..05539de Binary files /dev/null and b/reference-tables/v1/partitioned-parquet-data/data/a.parquet differ diff --git a/reference-tables/v1/partitioned-parquet-data/data/b.parquet b/reference-tables/v1/partitioned-parquet-data/data/b.parquet new file mode 100644 index 0000000..f4e361e Binary files /dev/null and b/reference-tables/v1/partitioned-parquet-data/data/b.parquet differ diff --git a/reference-tables/v1/partitioned-parquet-data/expected/scan.json b/reference-tables/v1/partitioned-parquet-data/expected/scan.json new file mode 100644 index 0000000..01d6bfa --- /dev/null +++ b/reference-tables/v1/partitioned-parquet-data/expected/scan.json @@ -0,0 +1,19 @@ +{ + "snapshots": [ + { + "snapshot-id": "1602692989443393359", + "rows": [ + {"1": "1", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"}, + {"1": "2", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"} + ] + }, + { + "snapshot-id": "5699608042525680694", + "rows": [ + {"1": "1", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"}, + {"1": "2", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"}, + {"1": "3", "2": "music", "3": "2026-01-02T00:00:00.000000+00:00"} + ] + } + ] +} diff --git a/reference-tables/v1/partitioned-parquet-data/metadata/516446c4-d70f-421a-af9a-50fe8654ebbd-m0.avro b/reference-tables/v1/partitioned-parquet-data/metadata/516446c4-d70f-421a-af9a-50fe8654ebbd-m0.avro new file mode 100644 index 0000000..9f355a6 Binary files /dev/null and b/reference-tables/v1/partitioned-parquet-data/metadata/516446c4-d70f-421a-af9a-50fe8654ebbd-m0.avro differ diff --git a/reference-tables/v1/partitioned-parquet-data/metadata/b0361de6-42e9-4841-ad9c-87a60d52d11d-m0.avro b/reference-tables/v1/partitioned-parquet-data/metadata/b0361de6-42e9-4841-ad9c-87a60d52d11d-m0.avro new file mode 100644 index 0000000..0b84382 Binary files /dev/null and b/reference-tables/v1/partitioned-parquet-data/metadata/b0361de6-42e9-4841-ad9c-87a60d52d11d-m0.avro differ diff --git a/reference-tables/v1/partitioned-parquet-data/metadata/snap-1602692989443393359-1-b0361de6-42e9-4841-ad9c-87a60d52d11d.avro b/reference-tables/v1/partitioned-parquet-data/metadata/snap-1602692989443393359-1-b0361de6-42e9-4841-ad9c-87a60d52d11d.avro new file mode 100644 index 0000000..4690158 Binary files /dev/null and b/reference-tables/v1/partitioned-parquet-data/metadata/snap-1602692989443393359-1-b0361de6-42e9-4841-ad9c-87a60d52d11d.avro differ diff --git a/reference-tables/v1/partitioned-parquet-data/metadata/snap-5699608042525680694-1-516446c4-d70f-421a-af9a-50fe8654ebbd.avro b/reference-tables/v1/partitioned-parquet-data/metadata/snap-5699608042525680694-1-516446c4-d70f-421a-af9a-50fe8654ebbd.avro new file mode 100644 index 0000000..e48cdae Binary files /dev/null and b/reference-tables/v1/partitioned-parquet-data/metadata/snap-5699608042525680694-1-516446c4-d70f-421a-af9a-50fe8654ebbd.avro differ diff --git a/reference-tables/v1/partitioned-parquet-data/metadata/v3.metadata.json b/reference-tables/v1/partitioned-parquet-data/metadata/v3.metadata.json new file mode 100644 index 0000000..33c264a --- /dev/null +++ b/reference-tables/v1/partitioned-parquet-data/metadata/v3.metadata.json @@ -0,0 +1 @@ +{"format-version":1,"table-uuid":"a79c6b9f-76a3-472e-a4ed-85ff532ccd11","location":"file:///tmp/iceberg-verification/reference-tables/v1/partitioned-parquet-data","last-updated-ms":1788557522647,"last-column-id":3,"schema":{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":true,"type":"long"},{"id":2,"name":"category","required":true,"type":"string"},{"id":3,"name":"event_ts","required":true,"type":"timestamptz"}]},"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":true,"type":"long"},{"id":2,"name":"category","required":true,"type":"string"},{"id":3,"name":"event_ts","required":true,"type":"timestamptz"}]}],"partition-spec":[{"name":"category","transform":"identity","source-id":2,"field-id":1000},{"name":"event_day","transform":"day","source-id":3,"field-id":1001}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"category","transform":"identity","source-id":2,"field-id":1000},{"name":"event_day","transform":"day","source-id":3,"field-id":1001}]}],"last-partition-id":1001,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"write.parquet.compression-codec":"zstd"},"current-snapshot-id":5699608042525680694,"refs":{"main":{"snapshot-id":5699608042525680694,"type":"branch"}},"snapshots":[{"snapshot-id":1602692989443393359,"timestamp-ms":1788557522539,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"0","manifests-replaced":"0","added-data-files":"1","added-records":"2","added-files-size":"1087","changed-partition-count":"1","total-records":"2","total-files-size":"1087","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v1/partitioned-parquet-data/metadata/snap-1602692989443393359-1-b0361de6-42e9-4841-ad9c-87a60d52d11d.avro","schema-id":0},{"snapshot-id":5699608042525680694,"parent-snapshot-id":1602692989443393359,"timestamp-ms":1788557522647,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"1","manifests-replaced":"0","added-data-files":"1","added-records":"1","added-files-size":"979","changed-partition-count":"1","total-records":"3","total-files-size":"2066","total-data-files":"2","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v1/partitioned-parquet-data/metadata/snap-5699608042525680694-1-516446c4-d70f-421a-af9a-50fe8654ebbd.avro","schema-id":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1788557522539,"snapshot-id":1602692989443393359},{"timestamp-ms":1788557522647,"snapshot-id":5699608042525680694}],"metadata-log":[{"timestamp-ms":1788557522406,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v1/partitioned-parquet-data/metadata/v1.metadata.json"},{"timestamp-ms":1788557522539,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v1/partitioned-parquet-data/metadata/v2.metadata.json"}]} \ No newline at end of file diff --git a/reference-tables/v2/nested-types-parquet-data/data/a.parquet b/reference-tables/v2/nested-types-parquet-data/data/a.parquet new file mode 100644 index 0000000..4ee9317 Binary files /dev/null and b/reference-tables/v2/nested-types-parquet-data/data/a.parquet differ diff --git a/reference-tables/v2/nested-types-parquet-data/expected/scan.json b/reference-tables/v2/nested-types-parquet-data/expected/scan.json new file mode 100644 index 0000000..a47dbac --- /dev/null +++ b/reference-tables/v2/nested-types-parquet-data/expected/scan.json @@ -0,0 +1,13 @@ +{ + "snapshots": [ + { + "snapshot-id": "9153710085634724558", + "rows": [ + {"1": "1", "2": {"5": 40.7128, "6": -74.006}, "3": ["east", "coastal"], "4": {"keys": ["pop"], "values": [8336817]}}, + {"1": "2", "2": null, "3": null, "4": null}, + {"1": "3", "2": {"5": 51.5074, "6": -0.1278}, "3": [], "4": {"keys": [], "values": []}}, + {"1": "4", "2": null, "3": ["west", null], "4": {"keys": ["pop"], "values": [null]}} + ] + } + ] +} diff --git a/reference-tables/v2/nested-types-parquet-data/metadata/35668469-d7a9-4d61-9da5-c65b78dd5407-m0.avro b/reference-tables/v2/nested-types-parquet-data/metadata/35668469-d7a9-4d61-9da5-c65b78dd5407-m0.avro new file mode 100644 index 0000000..9277972 Binary files /dev/null and b/reference-tables/v2/nested-types-parquet-data/metadata/35668469-d7a9-4d61-9da5-c65b78dd5407-m0.avro differ diff --git a/reference-tables/v2/nested-types-parquet-data/metadata/snap-9153710085634724558-1-35668469-d7a9-4d61-9da5-c65b78dd5407.avro b/reference-tables/v2/nested-types-parquet-data/metadata/snap-9153710085634724558-1-35668469-d7a9-4d61-9da5-c65b78dd5407.avro new file mode 100644 index 0000000..1b9e6c0 Binary files /dev/null and b/reference-tables/v2/nested-types-parquet-data/metadata/snap-9153710085634724558-1-35668469-d7a9-4d61-9da5-c65b78dd5407.avro differ diff --git a/reference-tables/v2/nested-types-parquet-data/metadata/v2.metadata.json b/reference-tables/v2/nested-types-parquet-data/metadata/v2.metadata.json new file mode 100644 index 0000000..77611c7 --- /dev/null +++ b/reference-tables/v2/nested-types-parquet-data/metadata/v2.metadata.json @@ -0,0 +1 @@ +{"format-version":2,"table-uuid":"be762ba3-cef6-4d04-81b4-033c7db9b38e","location":"file:///tmp/iceberg-verification/reference-tables/v2/nested-types-parquet-data","last-sequence-number":1,"last-updated-ms":1792616362118,"last-column-id":9,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":true,"type":"long"},{"id":2,"name":"location","required":false,"type":{"type":"struct","fields":[{"id":5,"name":"lat","required":true,"type":"double"},{"id":6,"name":"lon","required":true,"type":"double"}]}},{"id":3,"name":"tags","required":false,"type":{"type":"list","element-id":7,"element":"string","element-required":false}},{"id":4,"name":"props","required":false,"type":{"type":"map","key-id":8,"key":"string","value-id":9,"value":"int","value-required":false}}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"last-partition-id":999,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"write.parquet.compression-codec":"zstd"},"current-snapshot-id":9153710085634724558,"refs":{"main":{"snapshot-id":9153710085634724558,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":9153710085634724558,"timestamp-ms":1792616362084,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"0","manifests-replaced":"0","added-data-files":"1","added-records":"4","added-files-size":"2167","changed-partition-count":"1","total-records":"4","total-files-size":"2167","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v2/nested-types-parquet-data/metadata/snap-9153710085634724558-1-35668469-d7a9-4d61-9da5-c65b78dd5407.avro","schema-id":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1792616362084,"snapshot-id":9153710085634724558}],"metadata-log":[{"timestamp-ms":1792616360584,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v2/nested-types-parquet-data/metadata/v1.metadata.json"}]} \ No newline at end of file diff --git a/reference-tables/v2/partitioned-parquet-data/data/a.parquet b/reference-tables/v2/partitioned-parquet-data/data/a.parquet new file mode 100644 index 0000000..b6b28ca Binary files /dev/null and b/reference-tables/v2/partitioned-parquet-data/data/a.parquet differ diff --git a/reference-tables/v2/partitioned-parquet-data/data/b.parquet b/reference-tables/v2/partitioned-parquet-data/data/b.parquet new file mode 100644 index 0000000..08463b7 Binary files /dev/null and b/reference-tables/v2/partitioned-parquet-data/data/b.parquet differ diff --git a/reference-tables/v2/partitioned-parquet-data/expected/scan.json b/reference-tables/v2/partitioned-parquet-data/expected/scan.json new file mode 100644 index 0000000..693ceae --- /dev/null +++ b/reference-tables/v2/partitioned-parquet-data/expected/scan.json @@ -0,0 +1,19 @@ +{ + "snapshots": [ + { + "snapshot-id": "464121021900750417", + "rows": [ + {"1": "1", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"}, + {"1": "2", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"} + ] + }, + { + "snapshot-id": "8206603330787222691", + "rows": [ + {"1": "1", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"}, + {"1": "2", "2": "books", "3": "2026-01-01T00:00:00.000000+00:00"}, + {"1": "3", "2": "music", "3": "2026-01-02T00:00:00.000000+00:00"} + ] + } + ] +} diff --git a/reference-tables/v2/partitioned-parquet-data/metadata/a0a27bf6-f023-4840-838f-5513264f8deb-m0.avro b/reference-tables/v2/partitioned-parquet-data/metadata/a0a27bf6-f023-4840-838f-5513264f8deb-m0.avro new file mode 100644 index 0000000..1924ae3 Binary files /dev/null and b/reference-tables/v2/partitioned-parquet-data/metadata/a0a27bf6-f023-4840-838f-5513264f8deb-m0.avro differ diff --git a/reference-tables/v2/partitioned-parquet-data/metadata/b82906e8-8a4d-496c-8461-6420f7764946-m0.avro b/reference-tables/v2/partitioned-parquet-data/metadata/b82906e8-8a4d-496c-8461-6420f7764946-m0.avro new file mode 100644 index 0000000..865d0c3 Binary files /dev/null and b/reference-tables/v2/partitioned-parquet-data/metadata/b82906e8-8a4d-496c-8461-6420f7764946-m0.avro differ diff --git a/reference-tables/v2/partitioned-parquet-data/metadata/snap-464121021900750417-1-a0a27bf6-f023-4840-838f-5513264f8deb.avro b/reference-tables/v2/partitioned-parquet-data/metadata/snap-464121021900750417-1-a0a27bf6-f023-4840-838f-5513264f8deb.avro new file mode 100644 index 0000000..d7b92f3 Binary files /dev/null and b/reference-tables/v2/partitioned-parquet-data/metadata/snap-464121021900750417-1-a0a27bf6-f023-4840-838f-5513264f8deb.avro differ diff --git a/reference-tables/v2/partitioned-parquet-data/metadata/snap-8206603330787222691-1-b82906e8-8a4d-496c-8461-6420f7764946.avro b/reference-tables/v2/partitioned-parquet-data/metadata/snap-8206603330787222691-1-b82906e8-8a4d-496c-8461-6420f7764946.avro new file mode 100644 index 0000000..2bde251 Binary files /dev/null and b/reference-tables/v2/partitioned-parquet-data/metadata/snap-8206603330787222691-1-b82906e8-8a4d-496c-8461-6420f7764946.avro differ diff --git a/reference-tables/v2/partitioned-parquet-data/metadata/v3.metadata.json b/reference-tables/v2/partitioned-parquet-data/metadata/v3.metadata.json new file mode 100644 index 0000000..440b54f --- /dev/null +++ b/reference-tables/v2/partitioned-parquet-data/metadata/v3.metadata.json @@ -0,0 +1 @@ +{"format-version":2,"table-uuid":"89a49230-98bc-4b4f-b62c-9aa60d32713b","location":"file:///tmp/iceberg-verification/reference-tables/v2/partitioned-parquet-data","last-sequence-number":2,"last-updated-ms":1790699028599,"last-column-id":3,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":true,"type":"long"},{"id":2,"name":"category","required":true,"type":"string"},{"id":3,"name":"event_ts","required":true,"type":"timestamptz"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[{"name":"category","transform":"identity","source-id":2,"field-id":1000},{"name":"event_day","transform":"day","source-id":3,"field-id":1001}]}],"last-partition-id":1001,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"write.parquet.compression-codec":"zstd"},"current-snapshot-id":8206603330787222691,"refs":{"main":{"snapshot-id":8206603330787222691,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":464121021900750417,"timestamp-ms":1790699028377,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"0","manifests-replaced":"0","added-data-files":"1","added-records":"2","added-files-size":"1087","changed-partition-count":"1","total-records":"2","total-files-size":"1087","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v2/partitioned-parquet-data/metadata/snap-464121021900750417-1-a0a27bf6-f023-4840-838f-5513264f8deb.avro","schema-id":0},{"sequence-number":2,"snapshot-id":8206603330787222691,"parent-snapshot-id":464121021900750417,"timestamp-ms":1790699028599,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"1","manifests-replaced":"0","added-data-files":"1","added-records":"1","added-files-size":"979","changed-partition-count":"1","total-records":"3","total-files-size":"2066","total-data-files":"2","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v2/partitioned-parquet-data/metadata/snap-8206603330787222691-1-b82906e8-8a4d-496c-8461-6420f7764946.avro","schema-id":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1790699028377,"snapshot-id":464121021900750417},{"timestamp-ms":1790699028599,"snapshot-id":8206603330787222691}],"metadata-log":[{"timestamp-ms":1790699027227,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v2/partitioned-parquet-data/metadata/v1.metadata.json"},{"timestamp-ms":1790699028392,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v2/partitioned-parquet-data/metadata/v2.metadata.json"}]} \ No newline at end of file diff --git a/reference-tables/v2/simple-append-parquet-data/data/a.parquet b/reference-tables/v2/simple-append-parquet-data/data/a.parquet new file mode 100644 index 0000000..255a191 Binary files /dev/null and b/reference-tables/v2/simple-append-parquet-data/data/a.parquet differ diff --git a/reference-tables/v2/simple-append-parquet-data/data/b.parquet b/reference-tables/v2/simple-append-parquet-data/data/b.parquet new file mode 100644 index 0000000..699b5de Binary files /dev/null and b/reference-tables/v2/simple-append-parquet-data/data/b.parquet differ diff --git a/reference-tables/v2/simple-append-parquet-data/expected/scan.json b/reference-tables/v2/simple-append-parquet-data/expected/scan.json new file mode 100644 index 0000000..e286c16 --- /dev/null +++ b/reference-tables/v2/simple-append-parquet-data/expected/scan.json @@ -0,0 +1,20 @@ +{ + "snapshots": [ + { + "snapshot-id": "7973649908883454602", + "rows": [ + {"1": "1", "2": "alpha", "3": "2026-01-01T00:00:00.000000+00:00"}, + {"1": "2", "2": "bravo", "3": "2026-01-02T00:00:00.000000+00:00"} + ] + }, + { + "snapshot-id": "8192414647100221600", + "rows": [ + {"1": "1", "2": "alpha", "3": "2026-01-01T00:00:00.000000+00:00"}, + {"1": "2", "2": "bravo", "3": "2026-01-02T00:00:00.000000+00:00"}, + {"1": "3", "2": "charlie", "3": "2026-01-03T00:00:00.000000+00:00"}, + {"1": "4", "2": null, "3": "2026-01-04T00:00:00.000000+00:00"} + ] + } + ] +} diff --git a/reference-tables/v2/simple-append-parquet-data/metadata/286215d8-27c8-4975-b1f3-24b68b8bcf4d-m0.avro b/reference-tables/v2/simple-append-parquet-data/metadata/286215d8-27c8-4975-b1f3-24b68b8bcf4d-m0.avro new file mode 100644 index 0000000..42bbd0a Binary files /dev/null and b/reference-tables/v2/simple-append-parquet-data/metadata/286215d8-27c8-4975-b1f3-24b68b8bcf4d-m0.avro differ diff --git a/reference-tables/v2/simple-append-parquet-data/metadata/973f6e2b-06d8-4eb6-beb3-44cfcdee2233-m0.avro b/reference-tables/v2/simple-append-parquet-data/metadata/973f6e2b-06d8-4eb6-beb3-44cfcdee2233-m0.avro new file mode 100644 index 0000000..94eda3e Binary files /dev/null and b/reference-tables/v2/simple-append-parquet-data/metadata/973f6e2b-06d8-4eb6-beb3-44cfcdee2233-m0.avro differ diff --git a/reference-tables/v2/simple-append-parquet-data/metadata/snap-7973649908883454602-1-973f6e2b-06d8-4eb6-beb3-44cfcdee2233.avro b/reference-tables/v2/simple-append-parquet-data/metadata/snap-7973649908883454602-1-973f6e2b-06d8-4eb6-beb3-44cfcdee2233.avro new file mode 100644 index 0000000..5272b83 Binary files /dev/null and b/reference-tables/v2/simple-append-parquet-data/metadata/snap-7973649908883454602-1-973f6e2b-06d8-4eb6-beb3-44cfcdee2233.avro differ diff --git a/reference-tables/v2/simple-append-parquet-data/metadata/snap-8192414647100221600-1-286215d8-27c8-4975-b1f3-24b68b8bcf4d.avro b/reference-tables/v2/simple-append-parquet-data/metadata/snap-8192414647100221600-1-286215d8-27c8-4975-b1f3-24b68b8bcf4d.avro new file mode 100644 index 0000000..2bef31f Binary files /dev/null and b/reference-tables/v2/simple-append-parquet-data/metadata/snap-8192414647100221600-1-286215d8-27c8-4975-b1f3-24b68b8bcf4d.avro differ diff --git a/reference-tables/v2/simple-append-parquet-data/metadata/v3.metadata.json b/reference-tables/v2/simple-append-parquet-data/metadata/v3.metadata.json new file mode 100644 index 0000000..ede44f3 --- /dev/null +++ b/reference-tables/v2/simple-append-parquet-data/metadata/v3.metadata.json @@ -0,0 +1 @@ +{"format-version":2,"table-uuid":"b595109c-bf66-4e7f-ad07-cfa355127965","location":"file:///tmp/iceberg-verification/reference-tables/v2/simple-append-parquet-data","last-sequence-number":2,"last-updated-ms":1788557522209,"last-column-id":3,"current-schema-id":0,"schemas":[{"type":"struct","schema-id":0,"fields":[{"id":1,"name":"id","required":true,"type":"long"},{"id":2,"name":"data","required":false,"type":"string"},{"id":3,"name":"event_ts","required":false,"type":"timestamptz"}]}],"default-spec-id":0,"partition-specs":[{"spec-id":0,"fields":[]}],"last-partition-id":999,"default-sort-order-id":0,"sort-orders":[{"order-id":0,"fields":[]}],"properties":{"write.parquet.compression-codec":"zstd"},"current-snapshot-id":8192414647100221600,"refs":{"main":{"snapshot-id":8192414647100221600,"type":"branch"}},"snapshots":[{"sequence-number":1,"snapshot-id":7973649908883454602,"timestamp-ms":1788557521981,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"0","manifests-replaced":"0","added-data-files":"1","added-records":"2","added-files-size":"986","changed-partition-count":"1","total-records":"2","total-files-size":"986","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v2/simple-append-parquet-data/metadata/snap-7973649908883454602-1-973f6e2b-06d8-4eb6-beb3-44cfcdee2233.avro","schema-id":0},{"sequence-number":2,"snapshot-id":8192414647100221600,"parent-snapshot-id":7973649908883454602,"timestamp-ms":1788557522209,"summary":{"operation":"append","manifests-created":"1","manifests-kept":"1","manifests-replaced":"0","added-data-files":"1","added-records":"2","added-files-size":"1004","changed-partition-count":"1","total-records":"4","total-files-size":"1990","total-data-files":"2","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0","iceberg-version":"Apache Iceberg 1.11.0 (commit 6976e020b894f6a6777704df2b8c4458cb291ae9)"},"manifest-list":"file:/tmp/iceberg-verification/reference-tables/v2/simple-append-parquet-data/metadata/snap-8192414647100221600-1-286215d8-27c8-4975-b1f3-24b68b8bcf4d.avro","schema-id":0}],"statistics":[],"partition-statistics":[],"snapshot-log":[{"timestamp-ms":1788557521981,"snapshot-id":7973649908883454602},{"timestamp-ms":1788557522209,"snapshot-id":8192414647100221600}],"metadata-log":[{"timestamp-ms":1788557520390,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v2/simple-append-parquet-data/metadata/v1.metadata.json"},{"timestamp-ms":1788557522001,"metadata-file":"file:/tmp/iceberg-verification/reference-tables/v2/simple-append-parquet-data/metadata/v2.metadata.json"}]} \ No newline at end of file