[laxmanmamididd]

Which issue does this PR close?

Backport of #6900 to branch-1.1. Closes #6779 on this branch.

Rationale for this change

branch-1.1 has the bug fixed on main by #6900: a broadcast hash join drops matching rows when the build side emits sliced string or binary batches. Native DISTINCT, grouped aggregates and window operators emit such batches, and coalesceBroadcastBatches appends them with VectorSchemaRootAppender, which assumes offsets that start at zero.

We hit it on a production Spark job on Comet 1.1.0. Output was missing 24 of 724,405 rows. At an inner join on a 16-character string key, vanilla Spark produced 30,410 rows and CometBroadcastHashJoin produced 30,407 from identical inputs. With spark.comet.exec.broadcastHashJoin.enabled=false the output matched vanilla exactly.

What changes are included in this PR?

Cherry-pick of 66c63bd (#6900). The ffi.md hunk is placed above this branch's "Wrapper Object Lifecycle" heading because the heading differs from main.

How are these changes tested?

UtilsSuite passes on branch-1.1 with -Pspark-4.0 on JDK 17, and the new test serializeBatches rebases offsets that start past zero fails without the change to Utils.scala. CometJoinSuite needs the native library and has not been run locally; CI will run it.

🤖 Generated with Claude Code

[andygrove]

Thanks for picking this up, and for including the production numbers. I compared this with 66c63bd30 and the diff matches it line for line. The only difference is where the ffi.md hunk sits, and the description explains that.

For the record, branch-1.2 already has #6900 because it merged before the branch was cut, so no other release branch needs a backport.

When this merges, the squash message should end with (cherry picked from commit 66c63bd307b1b11fb684e6b89e9bb220f444de7c) so the pre-release check in backporting.md can match it. I'll add that at merge time.

This looks good to me once CI is green.

[comphead]

Broadcast solution is getting closer