Commit 1a35018
BF: Seek file data to 0 before every retry in request(), not just for retryable status codes
Previously, data.seek(0) was only called inside the retry_statuses/retry_if
branch, which handles specific HTTP response codes (429, 500-504, and
zarr-specific conditions). When a ConnectionError occurred mid-upload
(e.g., connection dropped during a multi-minute large zarr chunk transfer),
tenacity caught the exception and retried, but the file pointer was left at
whatever position the read was interrupted.
On retry, requests computed Content-Length as (file_size - current_position)
and sent only the tail of the file. S3 received partial data whose MD5
didn't match the Content-MD5 header (computed from the full file), resulting
in a BadDigest 400 error. This error was also not retried (not in
RETRY_STATUSES, not matched by retry_if), so the upload failed permanently.
The fix uses tenacity's before_sleep callback to seek the file data back to
position 0 before every retry, regardless of whether the retry was triggered
by ConnectionError, HTTPError, or a retryable status code.
Evidence from two independent logs (issue #1821):
- Linux: 188 level-0 zarr chunks failed (large, ~6 min upload each), 67
level-2 chunks succeeded (small, fast upload) -- all with "succeeded after
1 retry" + BadDigest
- Windows local disk: 187 level-0 failures after 2-8 retries, 68 level-1
successes -- ruling out NFS/filesystem as the cause
- Both logs show "Resetting dropped connection: dandiarchive.s3.amazonaws.com"
confirming ConnectionErrors preceded the BadDigest errors
Closes #1821
Co-Authored-By: Claude Code 2.1.81 / Claude Opus 4.6 <noreply@anthropic.com>1 parent 5f03d9b commit 1a35018
1 file changed
Lines changed: 9 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
212 | 212 | | |
213 | 213 | | |
214 | 214 | | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
215 | 223 | | |
216 | 224 | | |
217 | 225 | | |
| |||
225 | 233 | | |
226 | 234 | | |
227 | 235 | | |
| 236 | + | |
228 | 237 | | |
229 | 238 | | |
230 | 239 | | |
| |||
249 | 258 | | |
250 | 259 | | |
251 | 260 | | |
252 | | - | |
253 | | - | |
254 | 261 | | |
255 | 262 | | |
256 | 263 | | |
| |||
0 commit comments