Zarr output: write gridded and flat Zarr v3 stores

[robinskil]

Adds a write side to beacon-arrow-zarr, mirroring the NetCDF sinks so queries can be materialized as Zarr instead of only read from it. Both gridded (multi-dimensional) and flat output are supported.

What's here

• ZarrStoreWriter (writer.rs) builds a Zarr v3 store through zarrs' filesystem store, and packs it into a zip when the caller can only take one file.
• ZarrSink (sink.rs) writes flat output: each column a 1-D array over a shared obs dimension, appended as batches arrive so memory stays bounded.
• ZarrNdSink writes gridded output: the named dimension columns become coordinate arrays and the rest become N-D arrays indexed by them.
• ZarrFormat::create_writer_physical_plan picks between them based on ZarrOptions::write_dimensions.

Exposed as the zarr and ndzarr output formats, plus a writable format-factory registration so COPY TO ... STORED AS ZARR works. Mirrored in the Python CLI and TS SDK.

Delivery: zip vs directory

A zarr store is a directory, but an HTTP response carries one file. The query API therefore returns a zip (application/zip, .zarr.zip), readable via zarr-python's ZipStore or fsspec; entries are Stored (uncompressed) since chunk data is already codec-compressed. COPY TO has no such limit and writes a plain directory store.

Three decisions worth reviewing

Gridding does not reuse the NetCDF nd path. That path adds UniqueValuesExec + SortExec + SortPreservingMergeExec and scatters with ~300 lines of per-type macros. Here, cell positions come from a dense rank computed with stock Arrow kernels (sort_to_indices → cmp::distinct → accumulate) feeding arrow::compute::take — type-agnostic, ~60 lines, and no extra plan nodes, since DataSinkExec already requires a single input partition. This is a deliberate divergence from the netCDF precedent.

Only Zarr v3 is written. zarrs models v2 as a read-side format converted to v3 in memory and ships no v2 writer (MetadataConvertVersion offers only Default/V3), and Beacon's own reader discovers v3 stores only. Emitting v2 would mean hand-rolling .zgroup/.zarray JSON and C-order chunks outside zarrs, untestable through our own reader. ZarrOptions::zarr_version exists so adding it later is non-breaking.

Float nulls become NaN, not null. NaN is the conventional CF/zarr missing-value marker and xarray reads it as missing, but a round-trip through Beacon's reader surfaces NaN rather than null. No _FillValue attribute is written: it cannot express NaN as a JSON number, and the sentinels available for other types (0/false/"") collide with real data, so advertising them would mask real values.

Testing

936 workspace tests pass, 0 failures. New coverage:

• Flat store round-trips through Beacon's own reader (values, strings, timestamps).
• Multi-batch appends across a chunk boundary, where a partial chunk is re-read and extended.
• Grid scatter placement, including uncovered cells and rows with null dimension values.
• Zip entry paths are store-relative and /-separated, so the archive unpacks into a usable store.
• End-to-end query tests for zarr, ndzarr (asserts shape [2,2] and dimension_names), and COPY TO.

⚠️ The Python round-trip tests added to integration-tests/test_nd_formats.py (unzip → read_zarr) are not yet run — they need the container harness. The zip→reader path is verified only at the Rust level so far.

Loose end

beacon_core::query::CopyTo has no Zarr variant. It has no references outside its own definition, so it looks like dead code — happy to add it if it's actually live.

[codecov]

Codecov Report

❌ Patch coverage is 84.22819% with 141 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.15%. Comparing base (cb8a8e3) to head (6d4b3c7).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...eacon-file-formats/beacon-arrow-zarr/src/writer.rs	76.29%	87 Missing ⚠️
...e-formats/beacon-arrow-zarr/src/datafusion/sink.rs	88.47%	31 Missing ⚠️
...le-formats/beacon-arrow-zarr/src/datafusion/mod.rs	75.47%	13 Missing ⚠️
beacon-api/src/axum/client/query.rs	0.00%	8 Missing ⚠️
beacon-core/src/query/output.rs	90.47%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #331      +/-   ##
==========================================
+ Coverage   75.69%   76.15%   +0.46%     
==========================================
  Files         303      308       +5     
  Lines       40972    42523    +1551     
==========================================
+ Hits        31012    32382    +1370     
- Misses       9960    10141     +181     

Files with missing lines	Coverage Δ
beacon-core/src/query/from.rs	79.88% <100.00%> (ø)
beacon-core/src/query_result.rs	70.96% <100.00%> (+1.47%)	⬆️
beacon-core/src/runtime.rs	85.88% <100.00%> (+1.31%)	⬆️
beacon-data-lake/src/file_formats.rs	100.00% <100.00%> (ø)
...ormats/beacon-arrow-zarr/src/datafusion/options.rs	100.00% <100.00%> (ø)
beacon-core/src/query/output.rs	53.63% <90.47%> (+8.69%)	⬆️
beacon-api/src/axum/client/query.rs	6.94% <0.00%> (-0.41%)	⬇️
...le-formats/beacon-arrow-zarr/src/datafusion/mod.rs	83.41% <75.47%> (+0.08%)	⬆️
...e-formats/beacon-arrow-zarr/src/datafusion/sink.rs	88.47% <88.47%> (ø)
...eacon-file-formats/beacon-arrow-zarr/src/writer.rs	76.29% <76.29%> (ø)

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.