chore: upgrade DataFusion to 54.1.0

[paleolimbot]

Upgrades DataFusion from 52.5.0 to 54.1.0 with the corresponding Arrow, Parquet, object_store, GeoArrow, and Zarr versions. Includes initial trait, downcast, execution-plan, and metadata-cache API migrations.

A few larger updates:

• The Parquet reader now uses a "moreslizer" instead of an opener; however, conceptually this fits well with what we were doing already so the change is minimal.
• Logical serde was updated, so our placeholder approach had to be tweaked.
• A few metadata bugs were encountered, both worked around: a subquery getting optimized into a join borks on metadata (so I disabled/tracked) and one logical optimizer rule can't handle unnest + select (which is in our doctests for what you normally do with ST_Dump).

The big diff is mostly the lockfile for the examples.

[paleolimbot]

@zhangfengcdt Can you take a look at this one...I think it's ready for 👀 !

[jiayuasu]

Could this also be applied in new_from_context()? Passing a regular SessionContext still hits the Unnest bug. This test fails in push_down_leaf_projections; changing the constructor to SedonaContext::new() returns the expected two rows.

Repro: save as rust/sedona/tests/supplied_context_dump.rs, then run cargo test -p sedona --test supplied_context_dump.

#[tokio::test]
async fn supplied_context_dump() -> datafusion::error::Result<()> {
    use datafusion::{
        functions::core::expr_fn::get_field,
        prelude::{col, SessionContext},
    };
    use sedona::context::SedonaContext;

    let ctx = SedonaContext::new_from_context(SessionContext::new())?;
    let batches = ctx
        .sql("SELECT ST_Dump(ST_GeomFromText('MULTIPOINT (0 0, 1 1)')) AS dump")
        .await?
        .unnest_columns(&["dump"])?
        .select(vec![get_field(col("dump"), "geom").alias("geometry")])?
        .collect()
        .await?;
    assert_eq!(batches.iter().map(|b| b.num_rows()).sum::<usize>(), 2);
    Ok(())
}

[jiayuasu]

Could we add the zero-point guard to extract_chunk_stats() too? An empty LAZ file using point format 6 has a 477..477 chunk. Normal reading now works, but collect_statistics=true still fails with Requested range was invalid. The old object_store 0.12.4 accepted that empty range.

Repro: save as rust/sedona-pointcloud/tests/empty_laz_statistics.rs, then run cargo test -p sedona-pointcloud --test empty_laz_statistics. The final call fails; setting collect_statistics to false succeeds.

#[tokio::test]
async fn empty_laz_statistics() {
    use las::{point::Format, Builder, Writer};
    use object_store::{local::LocalFileSystem, path::Path, ObjectStoreExt};
    use sedona_pointcloud::las::{metadata::LasMetadataReader, options::LasOptions};

    let dir = tempfile::tempdir().unwrap();
    let path = dir.path().join("empty.laz");
    let mut builder = Builder::from((1, 4));
    builder.point_format = Format::new(6).unwrap();
    builder.point_format.is_compressed = true;
    let mut writer = Writer::from_path(&path, builder.into_header().unwrap()).unwrap();
    writer.close().unwrap();

    let store = LocalFileSystem::new();
    let location = Path::from_filesystem_path(&path).unwrap();
    let object = store.head(&location).await.unwrap();
    let metadata = LasMetadataReader::new(&store, &object)
        .fetch_metadata().await.unwrap();
    assert_eq!(metadata.header.number_of_points(), 0);

    let mut options = LasOptions::default();
    options.collect_statistics = true;
    LasMetadataReader::new(&store, &object)
        .with_options(options)
        .fetch_metadata()
        .await
        .unwrap();
}

[jiayuasu]

This revision still uses DataFusion 52.5, while the example now depends on 54.1. Could we regenerate the lockfile against a Sedona revision containing the upgrade?

From this PR checkout:

cd examples/sedonadb-rust
cargo tree --locked --depth 1 -p sedonadb-rust-example
cargo tree --locked --depth 1 -p sedona
cargo check --locked

The first tree shows DataFusion 54.1; the second shows 52.5. The example then passes a 54.1 Expr from col("name") to sort_by, which expects a 52.5 Expr (E0308). The ? calls also have incompatible DataFusionError types. CI rewrites the Sedona dependency to the PR head before building, so it does not test this committed lockfile.