Skip to content

API: Handle null partition values in ResidualEvaluator startsWith/notStartsWith - #18590

Open
raghavyadav01 wants to merge 1 commit into
apache:mainfrom
raghavyadav01:fix/residual-startswith-null
Open

raghavyadav01 wants to merge 1 commit into
apache:mainfrom
raghavyadav01:fix/residual-startswith-null

Conversation

@raghavyadav01

Copy link
Copy Markdown

Closes #18412

ResidualEvaluator dereferences the partition value in startsWith and notStartsWith without a null check, so computing a residual for a partition whose value is null throws NullPointerException:

java.lang.NullPointerException: Cannot invoke "String.startsWith(String)" because the return value of
"org.apache.iceberg.expressions.BoundReference.eval(org.apache.iceberg.StructLike)" is null
	at org.apache.iceberg.expressions.ResidualEvaluator$ResidualVisitor.startsWith(ResidualEvaluator.java:215)

ManifestFilterManager computes a residual for every live file in a manifest it rewrites, so a delete by row filter fails whenever the manifest also holds a file with a null partition value. Scans hit it too: with or(isNull(s), startsWith(s, "a")) the null-partition file survives planning through the isNull branch, and FileScanTask.residual() then throws.

Change

Treat a null partition value as not matching, so startsWith returns alwaysFalse() and notStartsWith returns alwaysTrue().

This keeps ResidualEvaluator consistent with Evaluator, which already evaluates startsWith as evalRes != null && ((String) evalRes).startsWith(...) and notStartsWith as its negation, and it mirrors the guard in and notIn received in #17014.

Tests

Added testStartsWithWithNullStringPartition and testNotStartsWithWithNullStringPartition to TestResiduals, following the shape of the existing testInWithNullStringPartition / testNotInWithNullStringPartition cases. Both fail with NullPointerException on main and pass with this change.

…StartsWith

ResidualEvaluator dereferenced the partition value in startsWith and
notStartsWith without a null check, so computing a residual for a partition
whose value is null threw NullPointerException.

ManifestFilterManager computes a residual for every live file in a manifest
it rewrites, so a delete by row filter failed whenever the manifest also held
a file with a null partition value. Scans hit it too: with
or(isNull(s), startsWith(s, "a")) the null-partition file survives planning
through the isNull branch and FileScanTask.residual() then throws.

Treat a null partition value as not matching, so startsWith returns
alwaysFalse() and notStartsWith returns alwaysTrue(). This matches Evaluator,
which already evaluates startsWith as `evalRes != null && ...` and
notStartsWith as its negation, and mirrors the guard in and notIn received
in apache#17014.

Fixes apache#18412

Signed-off-by: Raghvendra Yadav <raghavmnnit@gmail.com>
@github-actions github-actions Bot added the API label Oct 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

API: ResidualEvaluator throws NullPointerException for null partition values in startsWith/notStartsWith

1 participant