[cekees]

C1: the archive modes, as four modes of one writer

Stacked on #97 (retire XDMF); needs cekees/ymf#20 merged first, as #97 does.

The decision recorded in the roster (2026-10-23): C1's four pathways are four modes of one writer, and the modes are combinations of switches. This PR adds the switches to AR_base, names the modes, and covers the combinations.

mode	incremental .ymf	HDF5 driver	document	log in the .ymf	arrays inline
without-a-net	off	mpio	global	off	off
split (default)	on	mpio	global	off	off
kitchen-sink	on	mpio	global	on	off
ymf-only	on	none	global	off	on

A switch given explicitly (incremental=, driver=, document=, includeLog=, inline=) wins over the mode. parun and pro run take --archive-mode, --archive-driver and --archive-document; RunOptions carries them and NumericalSolution passes them through.

The drivers and the document layouts (C1.4, both halves)

• driver="serial": one .h5 that master alone opens; the other ranks gather their arrays to it. Needs no parallel HDF5 and writes the file the mpio driver writes: the datasets agree byte for byte at 1, 2 and 3 ranks in both dataset layouts. Cannot be hot started on several ranks, and refuses to.
• driver="per-rank": every rank writes its own <name>_p<rank>.h5 with plain HDF5 and no communication. Per-rank files hold per-rank datasets, so this driver turns global_sync off (and says so in the log). With it, document chooses where the grids are described:
  • "global" (the roster's layout B): one <name>.ymf whose every step is a spatial collection of the ranks' grids, each referencing its own rank's file; master gathers the grids and writes its file's metadata table, so readArchiveDomain(name) and gatherTimes rebuild the document from master's file alone.
  • "per-rank" (layout C): one <name>_p<rank>.ymf per rank, each step a single grid, each rank appending its own document with no collective step and no single point of failure.
  • Every rank's file is self-describing either way (its own metadata rows and attributes), so a per-rank hot start reads nothing but its own file.
• The writers test ar.writesDatasets instead of ar.hdfFile is not None, which is wrong on the ranks that hold no file.

The rest

• includeLog (C1.2): the run's log goes under extra: proteus in the document at close, with its file name and the command line. Every document, in every mode, carries proteus's version, the rank count, the mode and the driver (and the rank, for a per-rank document) from the first step's header on, so a killed run's document already says what wrote it. The command line is kitchen-sink only, on purpose: two runs of the same computation (pro run and parun, which a test compares byte for byte) leave identical documents.
• inline (ymf-only): master keeps the arrays by dataset name and at every sync replaces each reference in the step's grids with the values; the document is built a step at a time and appended like the others. A reference with no values behind it is refused.
• without-a-net is named and tested: no .ymf exists until close, and the document close writes equals, byte for byte, the one readArchiveDomain rebuilds from the .h5.
• One fix found by the serial driver: sync() took its two barriers only on ranks holding the file.

Covering the combinations

check_archive in test_archive_parallel_metadata.py is the integrity check every layout goes through: each document the run left validates; each collection has its two steps in the expected shape (one grid, or one per rank); every data item names a dataset that exists in the file it says, with the dimensions it declares (load_array on each); the files on disk are exactly the expected ones; and a per-rank-file global document is rebuilt from master's file. It runs over the single-file drivers × both dataset layouts × 1–3 ranks (12 runs) and the per-rank driver × both document layouts × 1–3 ranks (6 runs), plus the per-rank hot start, the per-rank running documents, the serial driver's datasets against mpio's, the modes and the invalid combinations refused by name. pro run --archive-mode kitchen-sink --archive-driver serial is checked end to end. The file is 83 tests.

🤖 Generated with Claude Code