fix: simplify IN lists containing NULL literals in EliminateFilter by raghavyadav01 · Pull Request #26548 · apache/datafusion

raghavyadav01 · 2026-10-21T05:04:46Z

Which issue does this PR close?

Rationale for this change

Under filter semantics only rows for which the predicate is TRUE are kept, so NULL rejects a row exactly like FALSE. That makes two rewrites sound for IN lists that hold NULL literals:

x IN (a, NULL) is TRUE exactly when x matches a. A NULL member can only turn FALSE into NULL, and both reject. The NULL members can be dropped, and a list of nothing but NULLs rejects every row.
x NOT IN (a, NULL) is FALSE when x matches a and NULL otherwise, so it is never TRUE and rejects every row.

Leaving the NULLs in place keeps a filter that PruningPredicate does not handle well, which is the pruning degradation described in the issue. #24711 fixed this only for the compact path (lists over 20 members) and #25187 made short lists on several types reach the same path.

What changes are included in this PR?

An Expr::InList arm in simplify_filter_predicate, plus a simplify_filter_in_list helper and an is_null_literal predicate, in datafusion/optimizer/src/eliminate_filter.rs.

The rewrite stays in EliminateFilter deliberately. It is not sound in ExprSimplifier: in a general expression context SELECT 3 IN (1, NULL) must remain NULL, and dropping the NULL would make it FALSE.

The negated case keeps the original InList inside FilterPredicate::RejectsAll rather than substituting a literal, so nothing is discarded that the AND/OR reconstruction may still need.

Are these changes tested?

Yes — five tests in eliminate_filter.rs:

test	asserts
`filter_in_list_with_null_drops_null`	`a IN (1, 2, NULL)` → `a IN (1, 2)`
`filter_in_list_of_only_nulls_rejects_all`	`a IN (NULL)` → `EmptyRelation`
`filter_not_in_list_with_null_rejects_all`	`a NOT IN (1, NULL)` → `EmptyRelation`
`filter_in_list_without_null_is_untouched`	no `NULL` → unchanged
`filter_not_in_list_without_null_is_untouched`	no `NULL` → unchanged

I wrote the tests first and confirmed the three NULL cases fail on main before the change and pass after, while the two control tests pass in both states.

Also run: the full datafusion-optimizer suite (948 + 29 + 5 passing), cargo fmt --all -- --check, and cargo clippy -p datafusion-optimizer --all-targets -- -D warnings, all clean. in_list.slt and the null_aware_* slt files pass. expr, predicates, scalar and aggregate slt files each fail identically with and without this change on my machine (a local test-data issue), so they are unaffected.

Are there any user-facing changes?

Explain output for filters whose IN list contains NULL literals: the NULL members no longer appear, and a filter that can never be TRUE becomes an empty relation. Query results are unchanged.

alamb

Thanks @raghavyadav01 -- I had a few suggestions

alamb · 2026-10-21T13:58:05Z

        Expr::BinaryExpr(binary) if binary.op == Operator::Or => {
            simplify_filter_or(*binary.left, *binary.right)
        }
+        Expr::InList(in_list) if in_list.list.iter().any(is_null_literal) => {


can we also update the comments to explain the new simplification?

Also rather than first checking is_null_literal, I think it could make sense to simply try the rewrite always

Done in b20f1f8 — the simplify_filter_predicate doc comment now points at the IN list case, and simplify_filter_in_list carries the reasoning and the reason it can't live in ExprSimplifier.

Good call — done. It now always attempts the rewrite and reports whether anything changed by comparing the list length before and after, which dropped the guard and collapsed the two "never TRUE" branches into one.

alamb · 2026-10-21T13:58:39Z

        ")
    }
+
+    #[test]


can you please add end to end .slt tests rather than unit tests here?

Done — unit tests removed and replaced with end-to-end tests in in_list.slt, covering the plans, the rows returned, and that SELECT 3 IN (1, NULL) is still NULL outside a filter.

Worth flagging what the slt tests showed, since it narrows the change: at SQL level only the NULL-dropping case actually differs. x IN (NULL) and x NOT IN (..., NULL) already reach EmptyRelation on main via existing simplification, so those tests are regression guards rather than new behaviour. I verified by reverting eliminate_filter.rs to origin/main — exactly one slt mismatch, the dropped NULL — and zero with the change.

I also kept the lists above THRESHOLD_INLINE_INLIST so they stay IN lists; below it they get rewritten into OR chains and the plan no longer shows what the test is about.

alamb

Thanks @raghavyadav01 -- this is looking close

alamb · 2026-10-23T15:40:35Z

+    } = in_list;
+
+    let len_before = list.len();
+    let list: Vec<Expr> = list.into_iter().filter(|e| !is_null_literal(e)).collect();


this makes a (deep) copy of each expression and I expect the common case will be that no entry is null -- can you please avoid making the copy before copying?

And then we can perhaps use Vec::retain to modify the existing vec rather than make a new copy

alamb · 2026-10-23T15:41:52Z

 # Cleanup
 statement ok
 DROP TABLE in_list_case_items;
+


Do you know why the plan that changed in https://github.com/apache/datafusion/pull/25187/changes#r4330163354 is not also changed in this PR?

I would exepect to see that plan simplified without the NULL in the list

Actually, looks like maybe it did fail:
https://github.com/raghavyadav01/datafusion/actions/runs/39909924986/job/120271950053

You may just have to update the branch and recapture slt results

`x IN (a, NULL)` is TRUE exactly when x matches a: a NULL member can only turn FALSE into NULL, and NULL rejects a row just like FALSE. The NULL members can therefore be dropped, and a list of nothing but NULLs rejects every row. `x NOT IN (a, NULL)` is FALSE when x matches a and NULL otherwise, so it is never TRUE and also rejects every row. Leaving the NULLs in place kept filters that PruningPredicate does not handle well, which is the degradation described in apache#26542. Extend simplify_filter_predicate with an Expr::InList arm. This is sound only under filter semantics, where just the TRUE rows are kept; the same rewrite in ExprSimplifier would change SELECT 3 IN (1, NULL) from NULL to FALSE, so it stays in EliminateFilter. Closes apache#26542

- Replace the unit tests with end-to-end tests in in_list.slt, covering the rewritten plans, the rows they return, and that SELECT 3 IN (1, NULL) is still NULL outside a filter. - Attempt the rewrite for every InList instead of guarding the match arm on is_null_literal; simplify_filter_in_list now reports whether it changed anything by comparing the list length. - Explain the new simplification in the simplify_filter_predicate doc comment. The slt lists are kept above THRESHOLD_INLINE_INLIST so they stay IN lists rather than being rewritten into OR chains.

`simplify_filter_in_list` built a fresh `Vec` on every call, including the common case where the list holds no `NULL` at all. `Vec::retain` drops them in place instead, so a list without `NULL`s costs one pass and no allocation. `eliminate_outer_join.slt` holds the other plan this rewrite reaches: select * from t1 left join t2 on t1.a = t2.x where t2.x in (1, 2, null) Both of its filters now read `IN ([Int32(1), Int32(2)])`. That is the same rewrite the PR is about - under filter semantics a `NULL` member can only turn `FALSE` into `NULL`, and both are rejected - so the rows are unchanged. The file came from apache#25187, which is the plan the review asked about. Rebased onto main. The full sqllogictest run fails in the same 236 places before and after. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Raghvendra Yadav <raghavmnnit@gmail.com>

raghavyadav01 · 2026-10-23T20:16:24Z

Both done.

retain drops the NULLs in place now, so a list with none — the usual case — doesn't allocate.

You were right about the #25187 plan. It's eliminate_outer_join.slt:95, and it was failing rather than updated. Rebased and recaptured — both filters read IN ([Int32(1), Int32(2)]) now. Full slt fails in the same 236 places before and after.

github-actions Bot added the optimizer Optimizer rules label Oct 21, 2026

alamb reviewed Oct 21, 2026

View reviewed changes

alamb added the performance Make DataFusion faster label Oct 21, 2026

github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Oct 22, 2026

alamb reviewed Oct 23, 2026

View reviewed changes

raghavyadav01 and others added 3 commits October 23, 2026 13:11

raghavyadav01 force-pushed the fix/eliminate-filter-in-list-null branch from 51de1b2 to 44da655 Compare October 23, 2026 20:16

Navigation Menu

Repository navigation

fix: simplify IN lists containing NULL literals in EliminateFilter - #26548

fix: simplify IN lists containing NULL literals in EliminateFilter#26548
raghavyadav01 wants to merge 3 commits into
apache:mainfrom
raghavyadav01:fix/eliminate-filter-in-list-null

raghavyadav01 commented Oct 21, 2026

Uh oh!

alamb left a comment

Uh oh!

alamb Oct 21, 2026

Uh oh!

alamb Oct 21, 2026

Uh oh!

raghavyadav01 Oct 22, 2026

Uh oh!

raghavyadav01 Oct 22, 2026

Uh oh!

alamb Oct 21, 2026

Uh oh!

raghavyadav01 Oct 22, 2026

Uh oh!

alamb left a comment

Uh oh!

alamb Oct 23, 2026

Uh oh!

alamb Oct 23, 2026

Uh oh!

alamb Oct 23, 2026

Uh oh!

raghavyadav01 commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

                       ")
                   }
+                  #[test]

Conversation

raghavyadav01 commented Oct 21, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

raghavyadav01 commented Oct 23, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants