Skip to content

[Variant] Normalize marked Variant arrays at the native Parquet boundary #5549

Description

@peterxcli

Problem

Once Variant identity exists in the native schema, ordinary Parquet can still return metadata, residual value, and optional typed_value in a physical Struct that is not Spark's required whole-value layout.

Parent: #5546
Depends on: #5548 (merged in #5631)
Implementation: #5715 (in review)
Related: #3983

Solution

The JVM scan gate remains closed in this task. It must not add subfield pruning, predicate pushdown, shredded writing, or JVM-side unshredding.

Tests

  • Canonical unshredded, fully shredded, and partially shredded values agree.
  • Inputs whose children are not in Spark output order are reordered.
  • Objects, arrays, scalars, JSON null, SQL NULL, nested residuals, and nullable parents round-trip.
  • typed_value never reaches the output.
  • The output Field marker and parent null bitmap are preserved.
  • Remove Variant UTF-16 output rewriting #5474 is linked to removable legacy ordering compatibility code.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:scanParquet scan / data readingenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions