[AdamGS]

Which issue does this PR close?

• Closes When performing set comparison subqueries (such as > ANY / > ALL / = ANY, etc.) on the same table (without using aliases), silently returns wrong results (0 rows or all rows). #25258.

Rationale for this change

During the various decorrelation passes, stripping outer references too early makes the outer and inner columns indistinguishable. Both operands are then rewritten to reference the subquery, turning the join predicate into a comparison of an inner column with itself (something like col1 > col1).

What changes are included in this PR?

Its mostly a minor change in where we strip the outer column reference from the join filter in the optimizer.

What is the testing strategy for this PR?

New SLT tests provided in the initial issue, asserting both results and logical plan shape.

Are there any user-facing changes?

Depreciating the find_join_exprs which isn't called internally anymore, replacing it with a helper function in decorrelate.rs.

[AdamGS]

I also suspect the mark join shape here is suboptimal, I'll follow up on that with a proper issue and hopefully a PR.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 93.65079% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.94%. Comparing base (79650ec) to head (0202867).
⚠️ Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/optimizer/src/decorrelate.rs	96.07%	0 Missing and 2 partials ⚠️
...on/optimizer/src/decorrelate_predicate_subquery.rs	83.33%	1 Missing ⚠️
...atafusion/optimizer/src/scalar_subquery_to_join.rs	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25273      +/-   ##
==========================================
- Coverage   82.94%   82.94%   -0.01%     
==========================================
  Files        1156     1156              
  Lines      458581   458623      +42     
  Branches   458581   458623      +42     
==========================================
+ Hits       380382   380399      +17     
- Misses      55145    55169      +24     
- Partials    23054    23055       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-expr v55.1.0 (current)
       Built [  38.190s] (current)
     Parsing datafusion-expr v55.1.0 (current)
      Parsed [   0.114s] (current)
    Building datafusion-expr v55.1.0 (baseline)
       Built [  32.314s] (baseline)
     Parsing datafusion-expr v55.1.0 (baseline)
      Parsed [   0.116s] (baseline)
    Checking datafusion-expr v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   2.269s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip

--- failure function_marked_deprecated: function #[deprecated] added ---

Description:
A function is now #[deprecated]. Downstream crates will get a compiler warning when using this function.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/function_marked_deprecated.ron

Failed in:
  function datafusion_expr::utils::find_join_exprs in /home/runner/work/datafusion/datafusion/datafusion/expr/src/utils.rs:1512

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  74.797s] datafusion-expr
    Building datafusion-optimizer v55.1.0 (current)
       Built [  30.769s] (current)
     Parsing datafusion-optimizer v55.1.0 (current)
      Parsed [   0.039s] (current)
    Building datafusion-optimizer v55.1.0 (baseline)
       Built [  30.707s] (baseline)
     Parsing datafusion-optimizer v55.1.0 (baseline)
      Parsed [   0.040s] (baseline)
    Checking datafusion-optimizer v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.246s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  63.433s] datafusion-optimizer

[AdamGS]

This function strips outer references which I think is somewhat unexpected, and was the root cause of this bug.
Its not called anymore, so deprecating it seems like a nice improvement, reducing the overall API surface.

[jayzhan211]

nit: this removes the outer-reference markers from the filters that are kept, not only for the comparison. So on the IN path they're gone again before build_join_filter gets to rename to the alias, which is the same issue this PR fixes. SQL can't hit it because inner columns shadow outer ones, but a hand-built plan can. Removing the markers from a copy keeps them intact, and the optimizer tests and subquery*.slt still pass with this:

     Ok(filters
         .into_iter()
-        .map(strip_outer_reference)
         .filter(|filter| {
+            let filter = &strip_outer_reference(filter.clone());
             if filter == in_predicate {

[jayzhan211]

nit: this function doesn't remove the markers any more, so "before stripping markers" reads a little oddly here. Maybe: "Compare with the outer-reference markers intact: inner and outer columns with the same qualified name belong to different scopes."

[jayzhan211]

Thanks @AdamGS , fix conflicts and nit are optional, then it is ready to merge

[AdamGS]

@jayzhan211 thank you for the review! I addressed both.