fix: don't proxy backend ETag by default - #23
Merged
Conversation
Some S3-compatible backends (e.g. VAST) return an ETag that is not a
true content MD5. Proxying it unconditionally broke AWS SDK/CLI
download integrity checks ("Unable to verify integrity of data
download"), including in N5/Zarr Java readers used by Fiji/BigDataViewer.
Add a per-target `proxy_etag` option (default false) to opt back in
for backends known to return a real content-MD5 ETag, like AWS S3.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
e45a732) started proxying the backend's rawETag/Last-Modifiedthrough on GetObject/HeadObject/ListObjectsV2. Some S3-compatible backends (e.g. VAST) return anETagthat isn't a true content MD5, which breaks the AWS SDK/CLI download integrity check ("Unable to verify integrity of data download"). This surfaced in production via N5/Zarr Java readers (Fiji/BigDataViewer), which use the AWS SDK v1 client that validates GetObject downloads against the ETag.proxy_etagoption onaiobototargets (defaultfalse) soETagis no longer proxied unless explicitly opted into for backends known to return a real content-MD5 ETag (e.g. real AWS S3).tests/java/src/test/java/org/janelia/x2s3/S3v1IntegrityTest.java, which reproduces the exact AWS SDK v1 code path that was throwing in Fiji, and updatesS3CompatTestto reflect the new default (ETag no longer proxied).Test plan
python -m pytest --cov=x2s3— 70 passedpixi run test-java(against a locally-running x2s3 pointed at real Janelia S3-gateway targets) — 16 passed, confirming noSdkClientExceptionand well-formed downloaded content@StephanPreibisch