[nicolamasotti]

The problem

DatasetCacher.cached_empty_dataset writes the .zarr schema sample of a request directly under its final name, abc.zarr. Every process that needs the same request uses the same name, so a process can open a sample that another one is still writing:

1. A and B need the sample for the same request, and neither finds abc.zarr.
2. A starts writing abc.zarr: first the group, then the coordinates, then the variable.
3. B finds abc.zarr already there and opens it, half-written.
4. B fails.

The error B gets depends on how far A has got:

• IndexError: list index out of range at client_cdsapi.py:156: no data variable written yet;
• KeyError: 't': a coordinate, such as surface, is taken as the variable;
• ContainsArrayError: An array exists in store ... at path 'step': B didn't find the folder either, and writes into it too.

Nothing in the code prevents this:

• The lock (get_write_lock(f"{HOSTNAME}-zarr")) is a thread lock in a hedp-data-access dask task (scheduler="threads"). It holds only among the threads of one process, so every dask worker has its own, even within one worker pod.
• The sample can be seen under its final name as soon as its first file is written.

Example 1: a hedp-data-access ingestion on the esa-hedp cluster

• What ran: the real setup.do_setup and ingestion.do_ingestion.
• The cluster: 2 worker pods, with 8 dask workers each (16 in total, --nthreads=1).
• The cache: one per worker pod (/tmp, shared by the pod's 8 dask workers), with cache_file: true.
• The data: a fake request client that writes real GRIB files.
• The setup: zarr chunks of 24 months, 5 dask tasks per batch, retries: 1.

Each dask task runs on its own dask worker and opens the dataset there. The dask workers can't see the sample the controller pod built during setup in its own /tmp, so 5 dask workers build it at the same time, each with its own lock.

request_chunks	One request covers	main	this PR
{"year": 10} (120 months)	5 zarr chunks	0 of 3 runs complete	3 of 3
{"year": 2} (24 months)	1 zarr chunk	0 of 3	3 of 3
{"year": 1} (12 months)	half a zarr chunk	0 of 3	3 of 3

On main, 8 of the 9 runs failed on a half-written sample and 1 failed with ContainsArrayError. With 1 dask task per batch, main completes in all three cases, because only one dask worker needs the sample at a time.

Example 2: a process killed while writing the sample

	main	this PR
Left in the cache	a half-written abc.zarr	abc.zarr.1a2b3c4d, which is never opened
Next runs	open it and fail, until it's deleted by hand	build a complete sample

The change

cached_empty_dataset now saves the sample the way GRIB downloads are saved, through robust_save_to_file:

	Writes to	Then
main	abc.zarr	—
this PR	abc.zarr.1a2b3c4d (random suffix)	renames it to abc.zarr. If another process got there first, it deletes its own copy and keeps theirs

So every process writes its own copy, the first one renamed into place is the one everybody opens, and abc.zarr only exists once it's complete.

robust_save_to_file also removes a temporary folder whose write failed. Before, it only removed files.

Costs

• Extra writes: processes that build the same sample at the same time each write their own copy, about 50 kB, and all but one throw it away. On main they also both wrote, into the same folder, or one opened the other's half-written sample and failed.
• Leftover folders: a process killed while writing leaves an abc.zarr.<random> folder behind. These can be removed with rm -r <cache_folder>/*.zarr.???????? when no run is active.

Not in this PR

A sample that is already broken under its final name is still used as it is. It may have been left by a process killed before this PR, or damaged when copied. A follow-up PR will handle it, introducing the possibility to overwrite the cached .zarr files.

Tests

New tests/test_30_engine_ecmwf.py:

Test	main	this PR
When another process saves the sample first, that one is kept and opened	fails	passes
A folder whose write failed is removed	fails	passes
A file that can't be renamed raises an error, and its temporary copy is removed	fails	passes

[dxnnystyles]

there's a race here for regular files. os.rename(tmp_path, path) can replace an existing destination on POSIX, so if two processes finish writing the same file, the second one can overwrite the first instead of preserving it. the Zarr directory case behaves differently. might be worth testing two writers targeting a regular file and using an atomic no-replace operation if first-writer-wins is the intended behavior for both.

[nicolamasotti]

It is true that on POSIX renaming a file onto an existing file replaces it. This doesn't happen with the .zarr sample, which is a directory: renaming a directory onto an existing sample fails, so the first sample saved is kept, and that's the case this PR handles. GRIB files, the only regular files saved this way, will be dealt with in a separate PR.