A zarr which works perfectly fine using the FileStore does not work when uploaded to S3. Here's an example:
https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/idr0066/ExpB_whole_cns_MIP.ome.zarr/0
Download it with ome_zarr (pip install ome-zarr), aws won't work because no List permission:
ome_zarr download https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/idr0066/ExpB_whole_cns_MIP.ome.zarr/0 --output ExpB_whole_cns_MIP.ome.zarr
Example code: https://gist.github.com/dominikl/3719fd5c3a44a159a68aec9d64724292
This will result in (I've just added some logging):
Testing local...
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
store: file:///Users/dom/testbf
group: <v3.Group {file:///Users/dom/testbf/ExpB_whole_cns_MIP.ome.zarr/0}>
First res: <v3.Array {file:///Users/dom/testbf/ExpB_whole_cns_MIP.ome.zarr/0/0} (1, 2, 1, 2738, 9357) UINT16>
Read array: 134 139 141 142 139 135 131 ...
Testing S3...
store: s3://idr/null
Handle: s3://idr/null/share/ome2024-ngff-challenge/idr0066/ExpB_whole_cns_MIP.ome.zarr/0
group: <v3.Group {s3://idr/null/share/ome2024-ngff-challenge/idr0066/ExpB_whole_cns_MIP.ome.zarr/0}>
First res: <v3.Array {s3://idr/null/share/ome2024-ngff-challenge/idr0066/ExpB_whole_cns_MIP.ome.zarr/0/0} (1, 2, 1, 2738, 9357) UINT16>
java.lang.RuntimeException: dev.zarr.zarrjava.ZarrException: The checksum of the sharding index is invalid. Stored: 524201981 Computed: 424531239
at dev.zarr.zarrjava.core.Array.lambda$1(Array.java:325)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
at dev.zarr.zarrjava.core.Array.read(Array.java:292)
at dev.zarr.zarrjava.core.Array.read(Array.java:233)
at com.example.zarrdemo.TestV3.loadArray(TestV3.java:81)
at com.example.zarrdemo.TestV3.testS3(TestV3.java:56)
at com.example.zarrdemo.TestV3.main(TestV3.java:37)
Caused by: dev.zarr.zarrjava.ZarrException: The checksum of the sharding index is invalid. Stored: 524201981 Computed: 424531239
at dev.zarr.zarrjava.v3.codec.core.Crc32cCodec.decode(Crc32cCodec.java:40)
at dev.zarr.zarrjava.core.codec.CodecPipeline.decode(CodecPipeline.java:113)
at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.decodeInternal(ShardingIndexedCodec.java:203)
at dev.zarr.zarrjava.v3.codec.core.ShardingIndexedCodec.decodePartial(ShardingIndexedCodec.java:253)
at dev.zarr.zarrjava.core.codec.CodecPipeline.decodePartial(CodecPipeline.java:94)
at dev.zarr.zarrjava.core.Array.lambda$1(Array.java:313)
I'm pretty stuck there unfortunately.
A zarr which works perfectly fine using the FileStore does not work when uploaded to S3. Here's an example:
https://uk1s3.embassy.ebi.ac.uk/idr/share/ome2024-ngff-challenge/idr0066/ExpB_whole_cns_MIP.ome.zarr/0
Download it with
ome_zarr(pip install ome-zarr), aws won't work because no List permission:Example code: https://gist.github.com/dominikl/3719fd5c3a44a159a68aec9d64724292
This will result in (I've just added some logging):
I'm pretty stuck there unfortunately.