From 23002e45ee5585162ee6803ac3959c7b3cdd056f Mon Sep 17 00:00:00 2001 From: damansingh1313 Date: Sun, 30 Aug 2026 15:01:36 +0530 Subject: [PATCH 1/4] docs: Document missing write table properties in configuration The write-options table omitted several properties that PyIceberg reads when writing, including write.target-file-size-bytes, write.format.default, write.avro.compression-codec, the Parquet row-group/bloom-filter sizes, the column metrics-mode properties and write.summary.partition-limit. Add rows for them and fix a typo ("coddec" -> "codec") in the existing compression-codec entry. --- mkdocs/docs/configuration.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mkdocs/docs/configuration.md b/mkdocs/docs/configuration.md index 18f9db973b..4f39931668 100644 --- a/mkdocs/docs/configuration.md +++ b/mkdocs/docs/configuration.md @@ -79,12 +79,21 @@ Iceberg tables support table properties to configure table behavior. | Key | Options | Default | Description | |------------------------------------------|------------------------------------|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| -| `write.parquet.compression-codec` | `{uncompressed,zstd,gzip,snappy}` | zstd | Sets the Parquet compression coddec. | +| `write.parquet.compression-codec` | `{uncompressed,zstd,gzip,snappy}` | zstd | Sets the Parquet compression codec. | | `write.parquet.compression-level` | Integer | null | Parquet compression level for the codec. If not set, it is up to PyIceberg | +| `write.parquet.row-group-size-bytes` | Size in bytes | 128MB | Target size of a Parquet row group when writing data files | | `write.parquet.row-group-limit` | Number of rows | 1048576 | The upper bound of the number of entries within a single row group | | `write.parquet.page-size-bytes` | Size in bytes | 1MB | Set a target threshold for the approximate encoded size of data pages within a column chunk | | `write.parquet.page-row-limit` | Number of rows | 20000 | Set a target threshold for the maximum number of rows within a column chunk | | `write.parquet.dict-size-bytes` | Size in bytes | 2MB | Set the dictionary page size limit per row group | +| `write.parquet.bloom-filter-max-bytes` | Size in bytes | 1MB | Maximum number of bytes for a Parquet bloom filter bitset per column | +| `write.parquet.bloom-filter-enabled.column.{column-name}` | Boolean | False | Enables writing a Parquet bloom filter for the given column | +| `write.target-file-size-bytes` | Size in bytes | 536870912 (512MB) | Target size of generated data files; larger inputs are split across multiple files | +| `write.format.default` | `{parquet}` | parquet | Default file format for data files written by PyIceberg | +| `write.avro.compression-codec` | `{gzip,zstd,snappy}` | gzip | Compression codec used when writing Avro files (for example, manifests) | +| `write.metadata.metrics.default` | `{none,counts,truncate(N),full}` | truncate(16) | Default column metrics mode recorded in manifests for data files | +| `write.metadata.metrics.column.{column-name}` | `{none,counts,truncate(N),full}` | (value of `write.metadata.metrics.default`) | Per-column override of the metrics mode | +| `write.summary.partition-limit` | Integer | 0 | Maximum number of partitions to include in a snapshot summary before the partition summary is omitted | | `write.metadata.previous-versions-max` | Integer | 100 | The max number of previous version metadata files to keep before deleting after commit. | | `write.metadata.delete-after-commit.enabled` | Boolean | False | Whether to automatically delete old *tracked* metadata files after each table commit. It will retain a number of the most recent metadata files, which can be set using property `write.metadata.previous-versions-max`. | | `write.object-storage.enabled` | Boolean | False | Enables the [`ObjectStoreLocationProvider`](configuration.md#object-store-location-provider) that adds a hash component to file paths. | From 8b5f5befb5d8732331b16572f4459d2a7e55498b Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Sat, 24 Oct 2026 10:50:18 -0700 Subject: [PATCH 2/4] docs: Remove unimplemented Parquet write properties PyIceberg ignores write.parquet.row-group-size-bytes and the bloom-filter properties, emitting a "not implemented" warning when they are set. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- mkdocs/docs/configuration.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/mkdocs/docs/configuration.md b/mkdocs/docs/configuration.md index 4f39931668..af0bc7aa49 100644 --- a/mkdocs/docs/configuration.md +++ b/mkdocs/docs/configuration.md @@ -81,13 +81,10 @@ Iceberg tables support table properties to configure table behavior. |------------------------------------------|------------------------------------|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| | `write.parquet.compression-codec` | `{uncompressed,zstd,gzip,snappy}` | zstd | Sets the Parquet compression codec. | | `write.parquet.compression-level` | Integer | null | Parquet compression level for the codec. If not set, it is up to PyIceberg | -| `write.parquet.row-group-size-bytes` | Size in bytes | 128MB | Target size of a Parquet row group when writing data files | | `write.parquet.row-group-limit` | Number of rows | 1048576 | The upper bound of the number of entries within a single row group | | `write.parquet.page-size-bytes` | Size in bytes | 1MB | Set a target threshold for the approximate encoded size of data pages within a column chunk | | `write.parquet.page-row-limit` | Number of rows | 20000 | Set a target threshold for the maximum number of rows within a column chunk | | `write.parquet.dict-size-bytes` | Size in bytes | 2MB | Set the dictionary page size limit per row group | -| `write.parquet.bloom-filter-max-bytes` | Size in bytes | 1MB | Maximum number of bytes for a Parquet bloom filter bitset per column | -| `write.parquet.bloom-filter-enabled.column.{column-name}` | Boolean | False | Enables writing a Parquet bloom filter for the given column | | `write.target-file-size-bytes` | Size in bytes | 536870912 (512MB) | Target size of generated data files; larger inputs are split across multiple files | | `write.format.default` | `{parquet}` | parquet | Default file format for data files written by PyIceberg | | `write.avro.compression-codec` | `{gzip,zstd,snappy}` | gzip | Compression codec used when writing Avro files (for example, manifests) | From 058ae567046a0369d9a0c0d866ff403af639b4e6 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Sat, 24 Oct 2026 10:50:19 -0700 Subject: [PATCH 3/4] docs: Clarify write table property descriptions Clarify how write.target-file-size-bytes is measured, what each metrics mode collects, which files the Avro codec applies to, and that the default write.summary.partition-limit of 0 disables partition summaries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- mkdocs/docs/configuration.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mkdocs/docs/configuration.md b/mkdocs/docs/configuration.md index af0bc7aa49..1330a596ab 100644 --- a/mkdocs/docs/configuration.md +++ b/mkdocs/docs/configuration.md @@ -85,12 +85,12 @@ Iceberg tables support table properties to configure table behavior. | `write.parquet.page-size-bytes` | Size in bytes | 1MB | Set a target threshold for the approximate encoded size of data pages within a column chunk | | `write.parquet.page-row-limit` | Number of rows | 20000 | Set a target threshold for the maximum number of rows within a column chunk | | `write.parquet.dict-size-bytes` | Size in bytes | 2MB | Set the dictionary page size limit per row group | -| `write.target-file-size-bytes` | Size in bytes | 536870912 (512MB) | Target size of generated data files; larger inputs are split across multiple files | -| `write.format.default` | `{parquet}` | parquet | Default file format for data files written by PyIceberg | -| `write.avro.compression-codec` | `{gzip,zstd,snappy}` | gzip | Compression codec used when writing Avro files (for example, manifests) | -| `write.metadata.metrics.default` | `{none,counts,truncate(N),full}` | truncate(16) | Default column metrics mode recorded in manifests for data files | -| `write.metadata.metrics.column.{column-name}` | `{none,counts,truncate(N),full}` | (value of `write.metadata.metrics.default`) | Per-column override of the metrics mode | -| `write.summary.partition-limit` | Integer | 0 | Maximum number of partitions to include in a snapshot summary before the partition summary is omitted | +| `write.target-file-size-bytes` | Size in bytes | 536870912 (512MB) | Target size of generated data files. Measured in uncompressed in-memory Arrow bytes, so files on disk are typically smaller after compression | +| `write.format.default` | `{parquet}` | parquet | File format for data files written by PyIceberg. Only Parquet is supported | +| `write.avro.compression-codec` | `{gzip,zstd,snappy}` | gzip | Compression codec for manifest files and manifest lists. For `snappy`, install with `pip install pyiceberg[snappy]` | +| `write.metadata.metrics.default` | `{none,counts,truncate(N),full}` | truncate(16) | Default column metrics for data files: `none` collects nothing, `counts` collects value and null counts, `full` adds lower/upper bounds, and `truncate(N)` truncates string and binary bounds to N characters or bytes | +| `write.metadata.metrics.column.{column-name}` | `{none,counts,truncate(N),full}` | (value of `write.metadata.metrics.default`) | Overrides `write.metadata.metrics.default` for a single column. Use the dot-separated name for nested fields | +| `write.summary.partition-limit` | Integer | 0 | Maximum number of changed partitions for which per-partition summaries are included in the snapshot summary. The default of 0 disables them | | `write.metadata.previous-versions-max` | Integer | 100 | The max number of previous version metadata files to keep before deleting after commit. | | `write.metadata.delete-after-commit.enabled` | Boolean | False | Whether to automatically delete old *tracked* metadata files after each table commit. It will retain a number of the most recent metadata files, which can be set using property `write.metadata.previous-versions-max`. | | `write.object-storage.enabled` | Boolean | False | Enables the [`ObjectStoreLocationProvider`](configuration.md#object-store-location-provider) that adds a hash component to file paths. | From ea0b57b8199a8d4595ed9f8063c7228438cee913 Mon Sep 17 00:00:00 2001 From: Kevin Liu Date: Sat, 24 Oct 2026 10:58:42 -0700 Subject: [PATCH 4/4] docs: Correct metrics mode description The none mode still records column sizes, and nested fields are capped at counts regardless of the configured mode. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --- mkdocs/docs/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mkdocs/docs/configuration.md b/mkdocs/docs/configuration.md index 1330a596ab..bd4470d085 100644 --- a/mkdocs/docs/configuration.md +++ b/mkdocs/docs/configuration.md @@ -88,7 +88,7 @@ Iceberg tables support table properties to configure table behavior. | `write.target-file-size-bytes` | Size in bytes | 536870912 (512MB) | Target size of generated data files. Measured in uncompressed in-memory Arrow bytes, so files on disk are typically smaller after compression | | `write.format.default` | `{parquet}` | parquet | File format for data files written by PyIceberg. Only Parquet is supported | | `write.avro.compression-codec` | `{gzip,zstd,snappy}` | gzip | Compression codec for manifest files and manifest lists. For `snappy`, install with `pip install pyiceberg[snappy]` | -| `write.metadata.metrics.default` | `{none,counts,truncate(N),full}` | truncate(16) | Default column metrics for data files: `none` collects nothing, `counts` collects value and null counts, `full` adds lower/upper bounds, and `truncate(N)` truncates string and binary bounds to N characters or bytes | +| `write.metadata.metrics.default` | `{none,counts,truncate(N),full}` | truncate(16) | Default column metrics for data files: `none` records only column sizes, `counts` adds value and null counts, `full` adds lower/upper bounds, and `truncate(N)` truncates string and binary bounds to N characters or bytes. Nested fields collect at most counts | | `write.metadata.metrics.column.{column-name}` | `{none,counts,truncate(N),full}` | (value of `write.metadata.metrics.default`) | Overrides `write.metadata.metrics.default` for a single column. Use the dot-separated name for nested fields | | `write.summary.partition-limit` | Integer | 0 | Maximum number of changed partitions for which per-partition summaries are included in the snapshot summary. The default of 0 disables them | | `write.metadata.previous-versions-max` | Integer | 100 | The max number of previous version metadata files to keep before deleting after commit. |