Skip to content

[C++][Parquet] Null value in nested non-nullable columns fails to write when the parent is null #55124

Open

Description

@HuaHuaY

Describe the bug, including details regarding any error messages, version, and platform.

auto type =
    ::arrow::struct_({::arrow::field("inner", ::arrow::int32(), /*nullable=*/false)});
auto values = ::arrow::ArrayFromJSON(type, R"([{"inner": 42}])");
auto indices = ::arrow::ArrayFromJSON(::arrow::int32(), "[0, null]");
ASSERT_OK_AND_ASSIGN(auto array, ::arrow::compute::Take(*values, *indices));

const auto& structs = checked_cast<const ::arrow::StructArray&>(*array);
ASSERT_EQ(structs.null_count(), 1);
ASSERT_EQ(structs.field(0)->null_count(), 1);
auto table = ::arrow::Table::Make(
    ::arrow::schema({::arrow::field("outer", type, /*nullable=*/true)}),
    {std::move(array)});
ASSERT_OK(table->ValidateFull());
CheckSimpleRoundtrip(table, /*row_group_size=*/2);

The Parquet writer checks the leaf array's physical null count without accounting for the parent struct's validity bitmap. As a result, it rejects nulls in required child fields even when they are masked by null parent rows.

// cpp/src/parquet/column_writer.cc

......

  Status WriteArrow(const int16_t* def_levels, const int16_t* rep_levels,
                    int64_t num_levels, const ::arrow::Array& leaf_array,
                    ArrowWriteContext* ctx, bool leaf_field_nullable) override {
    BEGIN_PARQUET_CATCH_EXCEPTIONS
    // Leaf nulls are canonical when there is only a single null element after a list
    // and it is at the leaf.
    bool single_nullable_element =
        (level_info_.def_level == level_info_.repeated_ancestor_def_level + 1) &&
        leaf_field_nullable;
    if (!leaf_field_nullable && leaf_array.null_count() != 0) {
      return Status::Invalid("Column '", descr_->name(),
                             "' is declared non-nullable but contains nulls");
    }

Component(s)

C++, Parquet

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions