[normanrz]

This PR adds a draft of RFC-16: Allowlist of Zarr extensions.

Right now the OME-Zarr spec allows any Zarr v3 data type, codec, chunk grid, chunk key encoding and storage transformer. This RFC replaces that with an allowlist that is part of the spec, to be in place before OME-Zarr 1.0. The list can be extended through the RFC process, including after 1.0.

Summary:

• Allowlist: a selection of the extensions in the Zarr v3 core spec and the zarr-extensions registry.
  • Data types: only the core bool, integer and floating-point types.
  • Codecs: core codecs plus registry codecs such as zstd, bitround, jpeg, zfp and scale_offset.
  • Chunk grids: regular and rectilinear.
  • Chunk key encodings: default, v2 and template.
  • Storage transformers: none.
• Reader levels: each entry is marked MUST, SHOULD or MAY. Readers must support the MUST entries. Writers should stick to them unless the data needs more.
• Extensions left out: listed with the reasons. They can be added later through an RFC.

Co-authored with @mkitti

[github-actions]

Automated Review URLs

• Readthedocs

[lubianat]

Thanks @normanrz and @mkitti !

As this is a new RFC, it will need to wait for Editorial Board approval before it can be merged.

I also noticed that it falls in a similar category as RFC 9 (https://ngff.openmicroscopy.org/rfc/9/index.html), where changes are somewhat orthogonal to the spec.

Instead of listing the codecs in the spec, would you consider a pointer to this RFC as the normative document?

Of the kind

All Zarr arrays in an OME-Zarr hierarchy MUST only use the data types, codecs, chunk grids, chunk key encodings and storage transformers listed in RFC-16.
This includes codecs nested in the configuration of other codecs.
Other sections of this specification MAY restrict the permitted data types further.
Readers MUST support the entries marked MUST, SHOULD support the entries marked SHOULD and MAY support the entries marked MAY.

And with future RFCs, the spec could be ammended to e.g.

All Zarr arrays in an OME-Zarr hierarchy MUST only use the data types, codecs, chunk grids, chunk key encodings and storage transformers listed in RFC-16 and RFC-XYZ.

In the hope to minimize churn/human errors & avoid drift between the RFC and the spec text, keeping the spec lean.

[normanrz]

I think it is useful that the spec remains self-contained without normative references to RFCs. I see the RFCs only as the vehicle for making changes to the spec, almost like a PR. And you wouldn't reference the PR description in a codebase.

That being said, it might make sense to break up the spec into multiple documents or appendices.

This seems to me an editorial decision. Happy to defer on that.

[imagesc-bot]

This pull request has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/ome-zarr-hackathon-how-to-represent-labels/123114/19

[clbarnes]

I see this RFC as along the lines of RFC-9 Zipped OME-Zarr - something which is more of a recommendation for OME-Zarr usage rather than a core part of the specification (an OME-Zarr Adoption Convention, if you will...). There we don't say that any other OME-Zarr in a zip is invalid, it's just a profile which we know works well and which implementations can target.

Whenever we're writing spec/ conventions we need to recognise that it's trivial for people to bend the spec and its implementations to their own use case. This is a key motivator for the extension proposal - people will stash their own metadata in there, so we should at least provide a mechanism for them to do so while still allowing their data to be read and not break on changes to OME-Zarr. IMO restricting codecs in particular is an invitation to do that - some use case will be served particularly well by some uncommon extension codec and if we say "you can't do that and still be OME-Zarr" I think the immediate response will be "OK, I'm not OME-Zarr, and now I don't have to care where else I break the spec".

I completely agree on having firm recommendations on which Zarr features to use when writing (with the warning that going outside those lines hampers data FAIRness), and which Zarr features readers should prioritise implementing, but I think further restrictions leaves us out of the strength that is Zarr extensibility.

[d-v-b]

I completely agree on having firm recommendations on which Zarr features to use when writing (with the warning that going outside those lines hampers data FAIRness), and which Zarr features readers should prioritise implementing, but I think further restrictions leaves us out of the strength that is Zarr extensibility.

seconding this. I don't see value in extending normative language to software, e.g. "Readers must support the MUST entries.". OME-Zarr should only use normative language to define the structure of stored data, i.e. the unit of interchange between applications. The behavior of a particular application (say, not supporting a common dtype) has no impact on interchange and so IMO doesn't need normative constraints.

edit: requiring that implementations refuse to read garbage / corrupted data is fine, but IMO that's an interchange concern (don't propagate invalid data) and very different from requiring that implementations support a fixed set of data types.