[andygrove]

Which issue does this PR close?

N/A. Related: #6410 and #7249.

Rationale for this change

Comet tracks the next DataFusion major version on a long-lived branch that builds against DataFusion main, so that breaks and regressions surface while there is still time to fix them upstream. The DataFusion 56 upgrade does this on the datafusion-56 branch (#7249, which replaces #6404 from a fork). The contributor guide doesn't describe the process, so the next upgrade would have to reconstruct it from old pull requests.

What changes are included in this PR?

A new contributor guide page, upgrading_datafusion.md, linked under "Project Mechanics". It covers:

• the datafusion-N branch in this repository, its draft pull request and the EPIC that tracks regressions
• pinning every DataFusion crate to one main revision, moving Arrow, Parquet and object_store with it, and pinning dependencies like iceberg-rust that must share the Arrow version
• fixing the build from DataFusion's upgrade guide, handling deprecations without a replacement, and checking DataFusion's behavior changes against Comet
• the CI labels to add, since the default checks don't run Spark's SQL tests, the Iceberg suites or the Delta contrib suites
• tracing each regression to the project that owns it, fixing it there, and ignoring and documenting a test when the fix won't land in time
• switching to the released crates and merging once DataFusion N.0.0 ships

How are these changes tested?

Documentation only. prettier --check passes on both changed files.

[sunchao]

Reviewed bb7c627963f74041ecad5b67db698cb79fe6ba0a across correctness of the documented procedure, compatibility, performance implications, design and validation. The dependency-pinning guidance, contrib build instructions and CI label names align with the repository, and current-head documentation CI is green.

There is one point to address before approval: the regression-handling instructions should require a fix or a safe fallback for newly incorrect results on enabled paths. Ignoring the test and documenting the limitation does not make those paths safe to ship. Details are inline.

[sunchao]

Could this distinguish a safely gated limitation from a new correctness regression on a default-enabled path? The preceding paragraph defines these as tests that passed on main. If the upgrade makes a native expression return a different result from Spark, ignoring that test and documenting the difference leaves the incorrect behavior enabled. Please require a fix, Spark fallback, or disabled/incompatible opt-in path before merging such a regression, and defer the upgrade when none is available. An issue link and an ignored test alone should not be enough to proceed to the merge step below.

[comphead]

The procedure matches the repository: the label names, the contrib-* features, the lockfiles and the linked anchors all exist at this head. One comment on how the shared branch is updated.

[comphead]

Resetting a branch that any committer can push to loses work in a way --force-with-lease does not catch. After a plain git fetch, the lease compares against the ref just fetched, so a commit another committer pushed in the meantime is dropped unless the person resetting carries it over on purpose. Could this section say how pushers coordinate, for example by announcing the reset on the EPIC and asking for pending commits to be rebased onto the new head? Another option is to update the branch by merging main and to squash only when the upgrade merges, which the squash merge already does for main.