Retry once when the batch signer returns a partial map

[stevevanhooser]

Closes #309.

The gap

BatchSignedUrlLookup.lookup treated "the map does not name this uid" as one thing. It is two:

1. Data drift — the file was never in this scope. Fall back per uid, warn once, move on.
2. A lagged index — the file IS in the scope on the server, but the batch endpoint's map has not caught up with a just-completed bulk upload. Falling back per uid here defeats the batch design; a 28,000-member series that hit this would cost 28,000 getFileDetails calls where the batch would have taken one.

Run 229 of test-cloud-api.yml on TEST_USER_1 / prod, on the merge commit that fixed #306 (#308), showed case 2: the batch answered signer_calls=1 with a 2-entry map for a 4-member series; the two missing uids each took a per-uid round trip. Retrying the same scope 37 minutes later on run_attempt: 2 returned the full 4-entry map, so the endpoint DOES eventually catch up — just not by the time waitForAllBulkUploads reports done.

The fix

When a cached scope answers with a URL but a subsequent uid in the same scope misses, drop the cache entry, sleep DEFAULT_PARTIAL_MAP_RETRY_SECONDS (500 ms), and re-fetch once. If the second map covers the uid, take it and every subsequent uid in that scope resolves from the refreshed cache. If not, warn and fall back exactly as today.

Bounded to one retry per scope, tracked in a new Stats.partial_map_retries counter — so a genuine data-drift miss still costs one extra signer call for the whole scope, not one per uid, and a run's own stats say whether the retry path fired.

A test seam (sleep= and partial_map_retry_seconds=) lets offline tests exercise the retry without paying the wall-clock cost.

Tests

Four new tests in TestBatchLookupPartialMapRetry:

• test_a_lagged_scope_settles_on_the_retry — first map has 2 of 4 uids, second has all 4. All four members resolve; signer_calls=2, partial_map_retries=1, uid_misses=0.
• test_the_retry_pauses_before_re_fetching — the 500 ms sleep fires exactly once, on the retry path.
• test_a_scope_is_only_retried_once — a scope whose map stays partial produces at most one retry regardless of how many misses follow.
• test_a_retry_that_returns_no_map_at_all_still_falls_back — retry raises → per-uid fallback, no double-count of the miss.

Existing test test_a_uid_absent_from_the_batch_map_falls_back updated to acknowledge the retry path: partial_map_retries=1, signer_calls=2, same fallback outcome. Retry sleep disabled in that test for speed.

Live-cloud round-trip test's assertion signer_calls == 1 relaxed to signer_calls == 1 + partial_map_retries, with a separate partial_map_retries <= 1 cap. The "one batch call per series' worth of members" contract survives; the retry is admitted.

Not changed

• filehandler.download_file_from_cloud — no change; it consults the same lookup.
• The signer itself — the fetch shape is untouched. The retry uses the existing _fetch_scope code path, populating the cache the same way.
• The negative-cache path (a signer that raised, or that returned an unusable payload) — still suppresses per-uid re-hammering exactly as before. The retry only fires when the FIRST answer was successful-but-partial.

Files

• src/ndi/cloud/batch_signed_url.py — retry logic + partial_map_retries stat.
• tests/test_cloud_batch_signed_url.py — new test class, one existing test refreshed.
• tests/test_cloud_file_series_round_trip.py — assertion admits one retry.

Offline tests pass (22 batch, 47 in the wider batch+series suite). Bridge check passes. The 7 pre-existing keyring failures in test_cloud_profile_secret_field_names.py reproduce on main; unrelated.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AdGZ8fRPeY9JSpUHgXm9ao

---

Generated by Claude Code