Skip to content

Spark: Fix NPE when reading/writing variant type with ORC format - #18387

Open
akashrn5 wants to merge 2 commits into
apache:mainfrom
akashrn5:NPE
Open

akashrn5 wants to merge 2 commits into
apache:mainfrom
akashrn5:NPE

Conversation

@akashrn5

@akashrn5 akashrn5 commented Oct 5, 2026 •

Copy link
Copy Markdown

Closes #18386

Summary

Writing a VARIANT column to a v3 table whose write format is ORC fails with a
NullPointerException from inside SparkOrcWriter, instead of the
UnsupportedOperationException("Variant is not supported") that
OrcSchemaWithTypeVisitor already defines for this case.

@github-actions github-actions Bot added the spark label Oct 5, 2026
@akashrn5 akashrn5 changed the title [WIP]Fix NPE in insert op for ORC format with variant type in v3 tables [WIP]Spark: Fix NPE in insert op for ORC format with variant type in v3 tables Oct 5, 2026
@akashrn5 akashrn5 changed the title [WIP]Spark: Fix NPE in insert op for ORC format with variant type in v3 tables Spark: Fix NPE in insert op for ORC format with variant type in v3 tables Oct 12, 2026
@akashrn5

Copy link
Copy Markdown
Author

Hi @nastra @uros-b can anyone please have a look at this PR? or if you can ask anyone else to review this. Thanks

Comment thread spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/data/TestOrcWrite.java Outdated
@akashrn5

Copy link
Copy Markdown
Author

@nastra can you please have a look at the changes and help to review and merge? i have handled all the review comments

File testFile = File.createTempFile("junit", null, temp.toFile());
assertThat(testFile.delete()).as("Delete should succeed").isTrue();

FileAppender<Record> writer =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we use a try-finally block here and maybe try and write a record?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i used try with resource and created an actual record for the test, please review

@nastra
nastra requested a review from huaxingao October 19, 2026 10:20
@nastra nastra changed the title Spark: Fix NPE in insert op for ORC format with variant type in v3 tables Spark: Fix NPE when reading/writing variant type with ORC format Oct 19, 2026
@nastra

nastra commented Oct 19, 2026

Copy link
Copy Markdown
Contributor

@akashrn5 can you please check whether we have the same issue with Flink's ORC reader/writer?

@nastra nastra closed this Oct 19, 2026
@nastra nastra reopened this Oct 19, 2026
@akashrn5

Copy link
Copy Markdown
Author

@akashrn5 can you please check whether we have the same issue with Flink's ORC reader/writer?

yes @nastra , i will raise a separate PR for flink instead of mixing it in spark module changes.

@uros-b

uros-b commented Oct 19, 2026

Copy link
Copy Markdown
Member

+1, thank you @akashrn5 and @nastra!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Spark/ORC] NullPointerException in SparkOrcWriter when inserting VARIANT data into ORC-backed Iceberg v3 tables

3 participants