You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Once Variant identity exists in the native schema, ordinary Parquet can still return metadata, residual value, and optional typed_value in a physical Struct that is not Spark's required whole-value layout.
Problem
Once Variant identity exists in the native schema, ordinary Parquet can still return
metadata, residualvalue, and optionaltyped_valuein a physical Struct that is not Spark's required whole-value layout.Parent: #5546
Depends on: #5548 (merged in #5631)
Implementation: #5715 (in review)
Related: #3983
Solution
VariantArrayby child name.unshred_variantonce at the native Parquet boundary.typed_valueand rebuild exactly[value, metadata]as ordinary Binary children.The JVM scan gate remains closed in this task. It must not add subfield pruning, predicate pushdown, shredded writing, or JVM-side unshredding.
Tests
typed_valuenever reaches the output.