Batch signed-URL lookup returns a partial map for a 4-member series on TEST_USER_1/prod #309

Surfaced by run 229 of test-cloud-api.yml after #308 merged. Three of the four (user × environment) cells pass; only TEST_USER_1 / prod fails, at the batch-signer efficiency assertion in test_members_survive_a_download_from_the_cloud:

AssertionError: a member fell back to per-uid getFileDetails.
Stats: signer_calls=1, uid_hits=2, uid_misses=2, failure_reason=''

last_map_size=2, last_map_uids=['41269720fdb9956d_40da7dec48b42117', '41269720fdb99570_c0d20060847e18de'].

What happened

The manifest and all 4 members uploaded and are on the server (that's #306, fixed in #308). All 4 members opened correctly on download — bytes match. What fails is the assertion that the batch signed-URL cache covers every uid in the scope: only 2 of 4 member uids came back in the batch map, so 2 members fell back to per-uid getFileDetails. Also emitted as a warning at src/ndi/cloud/filehandler.py:438:

The batch signed-URL lookup did not answer for scope "6aa931c04e21a03c972ae1c2/00065b84336133ad_4edab6a00cb0ac60/chunkdata.bin", so files there are being resolved one API call at a time. This still works, but for a large file series it is one call per member rather than one per series. Cause: the batch answered but did not name this uid. Reported once per scope.

Not a regression from #308

The download path and the batch-signer code (ndi/cloud/batch_signed_url.py, ndi/cloud/filehandler.py) weren't touched. But this is the first time a series document has ever successfully round-tripped on TEST_USER_1/prod — before #308, the manifest never reached the server (that's what run 227 caught), so this batch-signer behavior was hidden behind the upload gap.

Three of four (user × environment) cells produce a full map:

Two plausible causes

1. Timing/indexing race. waitForAllBulkUploads returns when the pipeline reports done, but the batch signed-URL endpoint on prod's cluster may lag behind — indexing 4 uids into the series scope isn't guaranteed to be transactional with the bulk-upload completion signal on all environments. A retry after a beat would settle it, and only User 1 / prod is slow enough to expose it.
2. Real gap on that account. A stale index, a per-account scope quirk, or a partial-response bug on the prod cluster specifically for User 1's datasets.

Next step

Re-run the failed job once to distinguish. If it passes on re-run, it's the timing case — likely fix on this side is a bounded retry in the batch signer's cache-population path, gated on stat coverage, before falling back to per-uid. If it fails again identically, it's an account/server issue and this issue tracks investigating the batch-signer scope query against the User 1 / prod cluster.

Refs

• Run 229 failure: https://github.com/Waltham-Data-Science/NDI-python/actions/runs/34965617453/job/104369237720
• Post-fix commit: f1f195f
• Test: tests/test_cloud_file_series_round_trip.py::TestFileSeriesRoundTrip::test_members_survive_a_download_from_the_cloud
• Sibling of uploadDataset silently skips a series manifest's bytes (uploaded doc references a nonexistent tempfile after DID ingests) #306 (the upload half), which Enumerate a document's binaries the way NDI-matlab does #308 closed.