Skip to content

Retry once when the batch signer returns a partial map - #992

Merged
stevewds merged 1 commit into
mainfrom
claude/batch-signer-partial-map-retry
Sep 15, 2026
Merged

stevewds merged 1 commit into
mainfrom
claude/batch-signer-partial-map-retry

Conversation

@stevevanhooser

Copy link
Copy Markdown
Contributor

Summary

  • Adds a bounded one-shot retry to ndi.cloud.download.internal.batchSignedUrlLookup for the case where the presign endpoint answers with a populated map that does not name a requested uid (a "partial map"). On the miss, the helper drops the entry, pauses briefly, and re-fetches once. If the second answer covers the uid, every subsequent uid in the same scope resolves from the refreshed cache; if not, the caller falls back per uid as before and the scope's warning fires. Bounded to one retry per scope, so a genuine data-drift miss still costs one extra signer call for the whole scope, not one per uid.
  • New STATS.partialMapRetries counter so tests can distinguish the retry path from a plain data-drift miss.
  • Refactor: the fetch/nargout/try dance moves into a nested localFetchScope so the retry can call it a second time without duplicating it. Adds an injectable sleepFcn (default @pause) and a partialMapRetrySeconds option so tests exercising the retry do not pay real wall-clock latency.
  • Relaxes FileSeriesRoundTripTest.signerCalls to 1 + batchStats.partialMapRetries with partialMapRetries <= 1, keeping the "one batch call per series' worth of members" contract while allowing the retry. uidMisses == 0 stays firm.

Motivation

Waltham-Data-Science/NDI-python#309 saw the second case on TEST_USER_1/prod after Waltham-Data-Science/NDI-python#308 first let a series manifest reach the server: the batch answered signer_calls=1 with a 2-entry map for a 4-member series. Re-running the same job 37 minutes later returned the full 4-entry map — so the batch endpoint does eventually catch up, just not by the time waitForAllBulkUploads reports done. The MATLAB helper has the same shape and the same risk. This PR is the MATLAB counterpart of Waltham-Data-Science/NDI-python#310, which merged the same fix on the Python side.

Closes #991.

Test plan

  • runtests('ndi.unittest.BatchSignedUrlLookupTest') — the four new testPartialMapRetry* tests cover retry-succeeds, retry-pauses, retry-once-per-scope, and retry-raises-and-falls-back; existing miss-path tests updated for the added signer call.
  • runtests('ndi.unittest.cloud.FileSeriesRoundTripTest') on live cloud — signerCalls == 1 + partialMapRetries and uidMisses == 0 hold across the previously flaky TEST_USER_1/prod matrix cell.

🤖 Generated with Claude Code

https://claude.ai/code/session_016rEFEXaZ9t3BftShqc9Y4B


Generated by Claude Code

ndi.cloud.download.internal.batchSignedUrlLookup collapsed two distinct
causes of "this uid is not in the scope's map" into one silent per-uid
fallback:

1. Data drift: the file was never in this scope. The right answer is to
   fall back and warn.
2. A lagged index: the file IS in the scope on the server, but the batch
   endpoint's map has not caught up with a just-completed bulk upload.
   Falling back per uid here defeats the batch design -- a
   28,000-member series that took the fallback would issue 28,000
   getFileDetails calls where the batch would have taken one.

Waltham-Data-Science/NDI-python#309 (the counterpart on the Python side)
observed the second case on TEST_USER_1/prod after
Waltham-Data-Science/NDI-python#308 first let a series manifest reach
the server: the batch answered signerCalls=1 with a 2-entry map for a
4-member series. The two missing uids fell back per uid. Retrying the
same scope 37 minutes later on run_attempt=2 returned the full 4-entry
map -- so the endpoint does eventually catch up, just not by the time
waitForAllBulkUploads reported done.

Fix (#991): when a cached scope answers with a URL but
a subsequent uid in that scope misses, drop the entry, pause for
partialMapRetrySeconds (default 0.5), and re-fetch once. If the second
map covers the uid, take it and every subsequent uid in the scope
resolves from the refreshed cache; if not, warn and fall back as
before. Bounded to one retry per scope, so a genuine data-drift miss
still costs one extra signer call for the whole scope, not one per
uid. Tracked in a new STATS.partialMapRetries counter so tests can
distinguish the retry path from a plain data-drift miss.

Refactor: the fetch/nargout/try dance moves into a nested
localFetchScope so the retry can call it a second time without
duplicating it. Adds an injectable sleepFcn (defaults to @Pause) so
tests exercising the retry path do not pay real wall-clock latency,
and a partialMapRetrySeconds=0 mode that keeps the single retry but
skips the sleep.

The live-cloud FileSeriesRoundTripTest.signerCalls assertion is
relaxed to signerCalls == 1 + partialMapRetries with
partialMapRetries <= 1, keeping the "one batch call per series' worth
of members" contract while allowing the retry. uidMisses == 0 stays
firm.

Four new unit tests in BatchSignedUrlLookupTest mirror the Python
counterpart's TestBatchLookupPartialMapRetry: the retry succeeds; the
retry pauses before re-fetching; the retry is bounded to once per
scope; a retry that raises still falls back cleanly without
double-counting the miss. Existing miss-path tests are updated to
account for the added signer call and to skip the sleep via
partialMapRetrySeconds=0.

Closes #991.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016rEFEXaZ9t3BftShqc9Y4B
@github-actions

Copy link
Copy Markdown
Contributor

Test Results

1 393 tests  +4   1 360 ✅ +4   3m 52s ⏱️ -59s
  206 suites ±0      33 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 9120c00. ± Comparison against base commit aea0c72.

@codecov

codecov Bot commented Sep 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.83333% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.85%. Comparing base (aea0c72) to head (9120c00).

Files with missing lines Patch % Lines
.../+cloud/+download/+internal/batchSignedUrlLookup.m 95.83% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #992      +/-   ##
==========================================
+ Coverage   29.79%   29.85%   +0.05%     
==========================================
  Files         857      857              
  Lines       43723    43750      +27     
==========================================
+ Hits        13029    13062      +33     
+ Misses      30694    30688       -6     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@stevewds
stevewds merged commit 16228d2 into main Sep 15, 2026
12 checks passed
@stevewds
stevewds deleted the claude/batch-signer-partial-map-retry branch September 15, 2026 18:46
stevevanhooser pushed a commit to Waltham-Data-Science/NDI-python that referenced this pull request Sep 15, 2026
NDI-matlab moved past 5804cd80 (VH-Lab/NDI-matlab#992 -- retry once
when the batch signer returns a partial map). Purely a hash bump to
satisfy test_matlab_bridge_hashes; the entry stays ported and no
Python-side behavior needs to follow yet.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

batchSignedUrlLookup: retry once when the presign endpoint returns a partial map

3 participants