fix: correct operator precedence for IS [NOT] DISTINCT FROM

[adriangb]

Which issue does this PR close?

• Closes type_coercion error: multi-condition IS NOT DISTINCT FROM in JOIN ON clause fails apache/datafusion#23692.

Rationale for this change

The issue reports this failing:

SELECT l.* FROM l LEFT ANTI JOIN r
  ON l.a IS NOT DISTINCT FROM r.a AND l.b IS NOT DISTINCT FROM r.b

Error during planning: Cannot infer common argument type for logical boolean operation Int64 AND Boolean

Neither the join nor the second condition is needed. The minimal reproducer is:

SELECT 1 IS NOT DISTINCT FROM 1 AND true

OR fails the same way, and so does every join type, WHERE, and a bare projection. What they have in common is a single IS [NOT] DISTINCT FROM with anything following it that binds less tightly.

The cause is not type coercion. sqlparser parses the right operand of IS [NOT] DISTINCT FROM with parse_expr() — that is, at the lowest possible precedence — instead of stopping at the first operator that binds less tightly than IS. Everything after the operator is swallowed into its right operand:

1 IS NOT DISTINCT FROM 1 AND true

  => IsNotDistinctFrom(1, BinaryOp { 1 AND true })

The Int64 AND Boolean in the error is that inner 1 AND true. PostgreSQL binds AND less tightly than IS, so the expected parse is (1 IS NOT DISTINCT FROM 1) AND true.

Parenthesising each condition is the workaround, which is why the bug went unnoticed: every existing IS NOT DISTINCT FROM test in the suite is parenthesised.

The greedy parse_expr() call is still present on datafusion-sqlparser-rs main (src/parser/mod.rs:4074), so the fix is applied on the DataFusion side. Fixing it upstream in the parser would be the cleaner home for it, and this workaround could then be dropped.

What changes are included in this PR?

In datafusion/sql/src/expr/mod.rs, before planning an expression the planner now restores the expected associativity:

1. has_greedy_distinct_from cheaply detects whether the mis-parse is present. When it isn't — the overwhelmingly common case — nothing else runs.
2. flatten_and_or flattens the AND / OR spine into operands and the operators separating them, re-attaching each IS [NOT] DISTINCT FROM to only the first operand of its right hand side.
3. rebuild_and_or rebuilds the expression with AND binding more tightly than OR, both left associative.

Prefix NOT is handled the same way, since it also binds more tightly than AND / OR. Handling OR and NOT is required for correctness rather than completeness: a purely local rotation gets a IS NOT DISTINCT FROM 1 AND b IS NOT DISTINCT FROM 2 OR c IS NOT DISTINCT FROM 3 wrong, producing A AND (B OR C) and turning a planning error into a silently wrong result.

Both helpers walk the spine iteratively, with explicit work stacks, for the same reason sql_expr_to_logical_expr uses a stack machine (apache#1444): they run in front of it, on every expression the planner sees, and deep AND / OR chains are common.

Operands are not descended into, so a parenthesised sub-expression keeps its explicit grouping and is handled when the planner recurses into it. The rewrite's output is a fixed point — it never leaves an IS [NOT] DISTINCT FROM whose right operand is an AND / OR — so re-entry cannot loop.

Known limitations left in place

• The postfix IS family at the same precedence level (a IS NOT DISTINCT FROM b IS NULL) is still associated as sqlparser produces it. That behaviour is unchanged by this PR and belongs with the broader precedence discussion in Operator precedence is inconsistent with modern PG (and PG 7.2) apache/datafusion#22461.
• A long chain of IS NOT DISTINCT FROM terms (~1024+) overflows the stack inside sqlparser's own recursive parse, before any DataFusion code runs — the same greedy parse_expr() makes each term nest in the AST instead of looping. Also not fixable from the planner side.

Are these changes tested?

Yes.

New planner tests in datafusion/sql/tests/sql_integration.rs covering join ON clauses, WHERE clauses, projections, IS DISTINCT FROM, AND / OR chains, and parenthesised forms as a control.

New end-to-end cases in datafusion/sqllogictest/test_files/select.slt for the minimal reproducer and the AND / OR / NOT cases around it, and in datafusion/sqllogictest/test_files/join_is_not_distinct_from.slt for the LEFT ANTI JOIN from the issue, three-condition chains, and the join-level precedence cases. The precedence cases use values where an incorrect grouping returns a different answer, not just a different plan — the plan display alone does not distinguish NOT (A AND B) from (NOT A) AND B.

New test_stack_overflow_distinct_from_{1024,8192} in datafusion/sql/src/expr/mod.rs, covering the fixup at the same spine depths the neighbouring test_stack_overflow tests use. As documented on the test, it is a scale check rather than a proof: these frames are small enough that a recursive walk would survive these depths too.

Verified locally:

• cargo test -p datafusion-sql — 89 + 564 + 12 doctests pass
• full sqllogictest suite — 501/501 files pass
• cargo test -p datafusion-optimizer -p datafusion-expr — passes
• cargo clippy -p datafusion-sql -p datafusion-optimizer --all-targets --all-features -- -D warnings — clean
• cargo fmt --all applied

Are there any user-facing changes?

Yes, and they are the point of the PR: IS [NOT] DISTINCT FROM combined with AND / OR / NOT without parentheses now parses the way PostgreSQL parses it, so queries that previously failed to plan now succeed.

Queries that were already parenthesised are unaffected. No public API changes.