diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f9fdf21a..e193f6f1 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -126,8 +126,9 @@ For the purposes of this discussion we classify features into the following buck the feature enabled can be read under an older version of the format, but some metadata might be missing or performance might be suboptimal. Simply phrased, forward compatible means all data can be read back in an older - version of the format. New logical types are considered forward - compatible despite the loss of semantic meaning. + version of the format. New logical types and new combinations of existing + logical/physical types are considered forward compatible despite the loss + of semantic meaning. 3. Forward incompatible. A file written under a newer version of the format with the feature enabled cannot be read under an older version of the format (e.g. diff --git a/LogicalTypes.md b/LogicalTypes.md index fae874c0..95969290 100644 --- a/LogicalTypes.md +++ b/LogicalTypes.md @@ -40,6 +40,8 @@ according to well defined conversion rules. ### Compatibility +#### ConvertedType + The Thrift definition of the metadata has two fields for logical types: `ConvertedType` and `LogicalType`. `ConvertedType` is an enum of all available annotations. Since Thrift enums can't have additional type parameters, it is cumbersome to define additional type parameters, like decimal scale and precision @@ -54,6 +56,13 @@ in case `LogicalType` annotations are not present. Parquet writers must always w `LogicalType` annotations where applicable, but must also write the corresponding `ConvertedType` annotations (if any) to maintain compatibility with old readers. +#### Unsupported Logical Types + +When reading a file written by a newer writer that contains an unrecognized logical type or an +unrecognized logical/physical type combination on a column, readers should ignore both the logical +type annotation and column order for that column. Only the physical type information should be used +to process the column's data. + Compatibility considerations are mentioned for each annotation in the corresponding section. ## String Types