feat: presigned URLs with extra query params and signed headers (SignedUrlOptions)#771
feat: presigned URLs with extra query params and signed headers (SignedUrlOptions)#771zfarrell wants to merge 12 commits into
Conversation
alamb
left a comment
There was a problem hiding this comment.
Thank you very much for this @zfarrell
I went through this PR carefully and I found it easy to read, and understand
I took the liberty of merging up from main and marking SignedUrlOptions as non_exhaustive to make potential future changes easier.
Blcoker: Testing
One blocker for me is I could not figure out how to run the new integration tests because:
- LocalStack doesn't validate pre-signed urls
- The bucket name is hard coded to
test-bucket-for-signingso I couldn't use an actual S3 account (I tried tor resolve 2 but it didn't work (see comment below))
Thus I am not sure how to verify this code works as designed.
Suggestion: End to End Doc Example of pre-signed multi-part uploads
One major thing I think would help here is an example in the docs, perhaps in S3::create_multipart, that shows an end to end example of how to implement multi-part signed urls (it doesn't have to make the actual client requests, of course); This PR includes several examples of doing this in the tests, but I think adding it to the docs will make it much more approachable. We (or I) can do this as a follow on.
| writes cannot contaminate the shared `test-bucket` whose exact contents `s3_test` asserts. Create | ||
| that bucket on whichever backend you point at before running them. | ||
|
|
||
| Run them against real S3, or against MinIO using the setup above: |
There was a problem hiding this comment.
If LocalStack can't run these tests, how did you test them?
I worry these tests will simply bitrot unless we run them regularly.
| ```shell | ||
| export TEST_INTEGRATION=1 | ||
| export TEST_S3_SIGNATURE_ENFORCEMENT=1 | ||
| cargo test --features aws --package object_store --lib aws::tests::signed_url -- --nocapture |
There was a problem hiding this comment.
I ran these commands, changing the hard coded bucket name:
diff --git a/src/aws/mod.rs b/src/aws/mod.rs
index 76b95cea..691c8a88 100644
--- a/src/aws/mod.rs
+++ b/src/aws/mod.rs
@@ -623,7 +623,7 @@ mod tests {
/// Presigned-URL tests run against a dedicated bucket so their concurrent writes cannot
/// contaminate the shared `test-bucket`, whose exact contents `s3_test` asserts.
- const SIGNING_BUCKET: &str = "test-bucket-for-signing";
+ const SIGNING_BUCKET: &str = "aal-test-bucket";
fn signing_store() -> AmazonS3 {
AmazonS3Builder::from_env()But they all failed 🤔
cargo test --features aws --package object_store --lib aws::tests::signed_url -- --nocapture
Compiling quick-xml v0.41.0
Compiling object_store v0.14.0 (/Users/andrewlamb/Software/arrow-rs-object-store)
Finished `test` profile [unoptimized + debuginfo] target(s) in 6.58s
Running unittests src/lib.rs (target/debug/deps/object_store-d310dbf2f9daf02f)
running 10 tests
thread 'aws::tests::signed_url_query_value_with_space' (136415050) panicked at src/aws/mod.rs:854:59:
called `Result::unwrap()` on an `Err` value: PermissionDenied { path: "test_signed_query_space.bin", source: RetryError(RetryErrorImpl { method: PUT, uri: Some(https://s3.us-east-1.amazonaws.com/aal-test-bucket/test_signed_query_space.bin), retries: 0, max_retries: 10, elapsed: 256.195125ms, retry_timeout: 180s, inner: Status { status: 403, body: Some("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>H7RWYCDEYN0N1WGZ</RequestId><HostId>BQux/WO9F3XyeMG6jsogf/PgfebIU0M7V8CQuCk/LO1c0PVxU4DIUPs1r76h6P/4Eg8PGI6sFu4W1C8JFwgxexON1NB2IfaW</HostId></Error>") } }) }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test aws::tests::signed_url_query_value_with_space ... FAILED
thread 'aws::tests::signed_url_special_character_key' (136415051) panicked at src/aws/mod.rs:1087:67:
called `Result::unwrap()` on an `Err` value: PermissionDenied { path: "test signed/a b=c/%CF%80.bin", source: RetryError(RetryErrorImpl { method: POST, uri: Some(https://s3.us-east-1.amazonaws.com/aal-test-bucket/test%20signed/a%20b%3Dc/%25CF%2580.bin?uploads=), retries: 0, max_retries: 10, elapsed: 155.700916ms, retry_timeout: 180s, inner: Status { status: 403, body: Some("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>H7RQVE2JYCY5SZQJ</RequestId><HostId>9rQuU1J1f3LsdLFa6/x9R79URTeqp/6yFlRH93jf84NetJwXVtl5mCFl/wbTJqpNawNZ4UzFMpwcjsYN82m1pWFDjMrfezk5</HostId></Error>") } }) }
test aws::tests::signed_url_special_character_key ... FAILED
thread 'aws::tests::signed_url_tampering_is_rejected' (136415052) panicked at src/aws/mod.rs:992:67:
called `Result::unwrap()` on an `Err` value: PermissionDenied { path: "test_signed_tamper.bin", source: RetryError(RetryErrorImpl { method: POST, uri: Some(https://s3.us-east-1.amazonaws.com/aal-test-bucket/test_signed_tamper.bin?uploads=), retries: 0, max_retries: 10, elapsed: 168.126125ms, retry_timeout: 180s, inner: Status { status: 403, body: Some("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>H7RJSP50QENMVTSF</RequestId><HostId>88jLCkKRWBnNOpvjKB7RrCIy3Ib3EgquWJHHpUsTff6++EZocjOpLTo/5lcqCLzTleRRtP0MB7rpTMVvQZaygIXM+9Smc+WE</HostId></Error>") } }) }
thread 'aws::tests::signed_url_with_signed_checksum_header_is_enforced' (136415053) panicked at src/aws/mod.rs:757:9:
matching checksum rejected: Response { url: "https://s3.us-east-1.amazonaws.com/aal-test-bucket/test_signed_checksum.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA5G3IGGKWACDYBBXE%2F20260710%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260710T133758Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host%3Bx-amz-checksum-sha256&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEO7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJIMEYCIQDAPOZaOTrE%2FM%2BpUBaVvbCQcHMj2epUtCpLabLyjWb1MQIhAPxAp%2FtcGK2Zd3hjQ5TEIKh2EBlhjoDv3VoMacpKneYCKvoDCLf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQABoMOTA4MDY3MjIyMTg4IgyZNKrjHSoGYBpLTC4qzgPHm2AoAMTVx4O3om9MyBe5W2l8Y30trRSBGfiJffGi1Fk6aJWK7jnY0E0sYYhg7qrGkmo4U0E6ktMCR2v3jn%2F3jBsMI53XHWkI8pfva2zgWcx65%2BkjbGgcQhTgRr2%2BMBFPgW75zfzNpoMNX7o%2B3fuVGI7NhMV7pTKfD%2BhxqAV23lH17E2KfNBRLpX4hTOLu8o4l7MHmwSU%2BtamTGIyoBgk2Y2ZU634balF7jpLKnQ%2FIIoTo4Joxor7bEMzUyoJjmCE%2BbUk4VMonc9DHkF3BR5c%2Bnidkg9iGCePsDDZDXeY%2BHPCH%2B8O4tXx8j%2Fd2DuHs2dCyzLs2gwsBpK7ydjhJ6xdqtw2HyU4HXah2zB6BfOEWTah8ZaF%2FGqFbduEg6GrYCp6JA8F2vZmibEyZevQDr%2BQ7oAvenkemPaTVaz%2FDhoPCaxbB3mR%2Fes6XgStGe9LhCHhZZrMMnz1AsdZtjUrVYbgXEaT%2FA6DE3QrmQrR3s2L2mkI0eHz1kCXiBlh7FBqIwA9SzFGAQeotuf9msKVAPKJ1av9Tdsq3JKtjeut%2B5q%2FEB4Rd7FCqB6f2MZBr9mJLC9Gg9OYKBL8JKujn6pCOx4Ot1ce5WMeesjZfk3LQMgw2%2BfD0gY6pwEEk2kEcW3SYLLHzawCNyHRLCFTRw3XpUmPJlFG92bg2aRa%2FCNlMZt7c9FaNfW7%2BLCM52kTQpNQdgiaxXL8srMSMe7GSW%2BQyxND5%2BhQchPGsNa92LbfaKFF4OOM7AZB358g2UkhYyDfvZxTjTgwQqVlLXQGIvRnZkYcxWaKsMzK%2BumpKyn1z0MuMxds%2BBWd5Tvoghom6hBZAESda6QWQQ8iPksDLvbT0w%3D%3D&X-Amz-Signature=a97ed1a4e53b099b460e46666a8908d27a7355b29b9fc0ef72d49d2b7804d845", status: 403, headers: {"x-amz-request-id": "H7RGT1MCJRWGD1XY", "x-amz-id-2": "eorVv3+7Pa1xn9sAnNSXbIZRAMUS5ccJOS81Un7ijKBS8Y8iUbO8lZNewZ0cqqMZopABgMwxqpE=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Fri, 10 Jul 2026 13:37:58 GMT", "connection": "close", "server": "AmazonS3"} }
test aws::tests::signed_url_tampering_is_rejected ... FAILED
test aws::tests::signed_url_with_signed_checksum_header_is_enforced ... FAILED
thread 'aws::tests::signed_url_multipart_multiple_parts_roundtrip' (136415048) panicked at src/aws/mod.rs:931:67:
called `Result::unwrap()` on an `Err` value: PermissionDenied { path: "test_signed_multipart_large.bin", source: RetryError(RetryErrorImpl { method: POST, uri: Some(https://s3.us-east-1.amazonaws.com/aal-test-bucket/test_signed_multipart_large.bin?uploads=), retries: 0, max_retries: 10, elapsed: 168.028792ms, retry_timeout: 180s, inner: Status { status: 403, body: Some("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>H7RQZ9SWW9KBAXDZ</RequestId><HostId>zlRGM+RSJWoZo3EyLR5S9f/t0GVDkZMbII9MtuPYVN6CtoXQMmKFed8UdmHXNUEhyvs2d69XbqF0dpg0phN869IbdrokxqlT</HostId></Error>") } }) }
test aws::tests::signed_url_multipart_multiple_parts_roundtrip ... FAILED
thread 'aws::tests::signed_url_multipart_upload' (136415049) panicked at src/aws/mod.rs:680:67:
called `Result::unwrap()` on an `Err` value: PermissionDenied { path: "test_signed_multipart_upload.bin", source: RetryError(RetryErrorImpl { method: POST, uri: Some(https://s3.us-east-1.amazonaws.com/aal-test-bucket/test_signed_multipart_upload.bin?uploads=), retries: 0, max_retries: 10, elapsed: 154.412542ms, retry_timeout: 180s, inner: Status { status: 403, body: Some("<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>H7RHT00VQ3P5Z3CM</RequestId><HostId>+87sbrEDhG6HMChQ6ZVKMexoE6Oi3M2PNwCB3IqWufB0fXQ1+YZzLtzBBFQMgNxQgeB63hRU2hY=</HostId></Error>") } }) }
test aws::tests::signed_url_multipart_upload ... FAILED
thread 'aws::tests::signed_url_with_signed_content_type_is_enforced' (136415054) panicked at src/aws/mod.rs:822:9:
matching content-type rejected: Response { url: "https://s3.us-east-1.amazonaws.com/aal-test-bucket/test_signed_content_type.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA5G3IGGKWACDYBBXE%2F20260710%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260710T133758Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-type%3Bhost&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEO7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJIMEYCIQDAPOZaOTrE%2FM%2BpUBaVvbCQcHMj2epUtCpLabLyjWb1MQIhAPxAp%2FtcGK2Zd3hjQ5TEIKh2EBlhjoDv3VoMacpKneYCKvoDCLf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQABoMOTA4MDY3MjIyMTg4IgyZNKrjHSoGYBpLTC4qzgPHm2AoAMTVx4O3om9MyBe5W2l8Y30trRSBGfiJffGi1Fk6aJWK7jnY0E0sYYhg7qrGkmo4U0E6ktMCR2v3jn%2F3jBsMI53XHWkI8pfva2zgWcx65%2BkjbGgcQhTgRr2%2BMBFPgW75zfzNpoMNX7o%2B3fuVGI7NhMV7pTKfD%2BhxqAV23lH17E2KfNBRLpX4hTOLu8o4l7MHmwSU%2BtamTGIyoBgk2Y2ZU634balF7jpLKnQ%2FIIoTo4Joxor7bEMzUyoJjmCE%2BbUk4VMonc9DHkF3BR5c%2Bnidkg9iGCePsDDZDXeY%2BHPCH%2B8O4tXx8j%2Fd2DuHs2dCyzLs2gwsBpK7ydjhJ6xdqtw2HyU4HXah2zB6BfOEWTah8ZaF%2FGqFbduEg6GrYCp6JA8F2vZmibEyZevQDr%2BQ7oAvenkemPaTVaz%2FDhoPCaxbB3mR%2Fes6XgStGe9LhCHhZZrMMnz1AsdZtjUrVYbgXEaT%2FA6DE3QrmQrR3s2L2mkI0eHz1kCXiBlh7FBqIwA9SzFGAQeotuf9msKVAPKJ1av9Tdsq3JKtjeut%2B5q%2FEB4Rd7FCqB6f2MZBr9mJLC9Gg9OYKBL8JKujn6pCOx4Ot1ce5WMeesjZfk3LQMgw2%2BfD0gY6pwEEk2kEcW3SYLLHzawCNyHRLCFTRw3XpUmPJlFG92bg2aRa%2FCNlMZt7c9FaNfW7%2BLCM52kTQpNQdgiaxXL8srMSMe7GSW%2BQyxND5%2BhQchPGsNa92LbfaKFF4OOM7AZB358g2UkhYyDfvZxTjTgwQqVlLXQGIvRnZkYcxWaKsMzK%2BumpKyn1z0MuMxds%2BBWd5Tvoghom6hBZAESda6QWQQ8iPksDLvbT0w%3D%3D&X-Amz-Signature=7235bfdec0fd593d9b439a1ff4e947b5e5d68620a6b57c6954d31c6420b2d143", status: 403, headers: {"x-amz-request-id": "H7RH83MKWPC4SS5N", "x-amz-id-2": "6OjHB+a8cBtXdVyHQWCoL3AsX5E1yajK/jA78f0le2rwiHMjEsVNuxa0PEN4Vye5SY2J3cpz5tI=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Fri, 10 Jul 2026 13:37:58 GMT", "connection": "close", "server": "AmazonS3"} }
test aws::tests::signed_url_with_signed_content_type_is_enforced ... FAILED
thread 'aws::tests::signed_url_baseline_roundtrip_without_options' (136415045) panicked at src/aws/mod.rs:904:9:
baseline PUT failed: Response { url: "https://s3.us-east-1.amazonaws.com/aal-test-bucket/test_signed_baseline.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA5G3IGGKWACDYBBXE%2F20260710%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260710T133758Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEO7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJIMEYCIQDAPOZaOTrE%2FM%2BpUBaVvbCQcHMj2epUtCpLabLyjWb1MQIhAPxAp%2FtcGK2Zd3hjQ5TEIKh2EBlhjoDv3VoMacpKneYCKvoDCLf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQABoMOTA4MDY3MjIyMTg4IgyZNKrjHSoGYBpLTC4qzgPHm2AoAMTVx4O3om9MyBe5W2l8Y30trRSBGfiJffGi1Fk6aJWK7jnY0E0sYYhg7qrGkmo4U0E6ktMCR2v3jn%2F3jBsMI53XHWkI8pfva2zgWcx65%2BkjbGgcQhTgRr2%2BMBFPgW75zfzNpoMNX7o%2B3fuVGI7NhMV7pTKfD%2BhxqAV23lH17E2KfNBRLpX4hTOLu8o4l7MHmwSU%2BtamTGIyoBgk2Y2ZU634balF7jpLKnQ%2FIIoTo4Joxor7bEMzUyoJjmCE%2BbUk4VMonc9DHkF3BR5c%2Bnidkg9iGCePsDDZDXeY%2BHPCH%2B8O4tXx8j%2Fd2DuHs2dCyzLs2gwsBpK7ydjhJ6xdqtw2HyU4HXah2zB6BfOEWTah8ZaF%2FGqFbduEg6GrYCp6JA8F2vZmibEyZevQDr%2BQ7oAvenkemPaTVaz%2FDhoPCaxbB3mR%2Fes6XgStGe9LhCHhZZrMMnz1AsdZtjUrVYbgXEaT%2FA6DE3QrmQrR3s2L2mkI0eHz1kCXiBlh7FBqIwA9SzFGAQeotuf9msKVAPKJ1av9Tdsq3JKtjeut%2B5q%2FEB4Rd7FCqB6f2MZBr9mJLC9Gg9OYKBL8JKujn6pCOx4Ot1ce5WMeesjZfk3LQMgw2%2BfD0gY6pwEEk2kEcW3SYLLHzawCNyHRLCFTRw3XpUmPJlFG92bg2aRa%2FCNlMZt7c9FaNfW7%2BLCM52kTQpNQdgiaxXL8srMSMe7GSW%2BQyxND5%2BhQchPGsNa92LbfaKFF4OOM7AZB358g2UkhYyDfvZxTjTgwQqVlLXQGIvRnZkYcxWaKsMzK%2BumpKyn1z0MuMxds%2BBWd5Tvoghom6hBZAESda6QWQQ8iPksDLvbT0w%3D%3D&X-Amz-Signature=20cc0b720ec6119c0042497096a3a102e954fe84a4fe4aeda86c5cfa269b3f6c", status: 403, headers: {"x-amz-request-id": "H7RH26BBCA1YED7X", "x-amz-id-2": "LydiARNNMi6FeCsKu00JbNbCCDhWnk/IK7XB171y8ry/2gK83ATFA1WfwLCNAxaF/jiyVfY5TtE=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Fri, 10 Jul 2026 13:37:58 GMT", "connection": "close", "server": "AmazonS3"} }
test aws::tests::signed_url_baseline_roundtrip_without_options ... FAILED
thread 'aws::tests::signed_url_conditional_create_blocks_overwrite' (136415046) panicked at src/aws/mod.rs:1183:9:
conditional create on absent object failed: Response { url: "https://s3.us-east-1.amazonaws.com/aal-test-bucket/test_signed_conditional.bin?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIA5G3IGGKWACDYBBXE%2F20260710%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260710T133758Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host%3Bif-none-match&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEO7%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEaCXVzLWVhc3QtMSJIMEYCIQDAPOZaOTrE%2FM%2BpUBaVvbCQcHMj2epUtCpLabLyjWb1MQIhAPxAp%2FtcGK2Zd3hjQ5TEIKh2EBlhjoDv3VoMacpKneYCKvoDCLf%2F%2F%2F%2F%2F%2F%2F%2F%2F%2FwEQABoMOTA4MDY3MjIyMTg4IgyZNKrjHSoGYBpLTC4qzgPHm2AoAMTVx4O3om9MyBe5W2l8Y30trRSBGfiJffGi1Fk6aJWK7jnY0E0sYYhg7qrGkmo4U0E6ktMCR2v3jn%2F3jBsMI53XHWkI8pfva2zgWcx65%2BkjbGgcQhTgRr2%2BMBFPgW75zfzNpoMNX7o%2B3fuVGI7NhMV7pTKfD%2BhxqAV23lH17E2KfNBRLpX4hTOLu8o4l7MHmwSU%2BtamTGIyoBgk2Y2ZU634balF7jpLKnQ%2FIIoTo4Joxor7bEMzUyoJjmCE%2BbUk4VMonc9DHkF3BR5c%2Bnidkg9iGCePsDDZDXeY%2BHPCH%2B8O4tXx8j%2Fd2DuHs2dCyzLs2gwsBpK7ydjhJ6xdqtw2HyU4HXah2zB6BfOEWTah8ZaF%2FGqFbduEg6GrYCp6JA8F2vZmibEyZevQDr%2BQ7oAvenkemPaTVaz%2FDhoPCaxbB3mR%2Fes6XgStGe9LhCHhZZrMMnz1AsdZtjUrVYbgXEaT%2FA6DE3QrmQrR3s2L2mkI0eHz1kCXiBlh7FBqIwA9SzFGAQeotuf9msKVAPKJ1av9Tdsq3JKtjeut%2B5q%2FEB4Rd7FCqB6f2MZBr9mJLC9Gg9OYKBL8JKujn6pCOx4Ot1ce5WMeesjZfk3LQMgw2%2BfD0gY6pwEEk2kEcW3SYLLHzawCNyHRLCFTRw3XpUmPJlFG92bg2aRa%2FCNlMZt7c9FaNfW7%2BLCM52kTQpNQdgiaxXL8srMSMe7GSW%2BQyxND5%2BhQchPGsNa92LbfaKFF4OOM7AZB358g2UkhYyDfvZxTjTgwQqVlLXQGIvRnZkYcxWaKsMzK%2BumpKyn1z0MuMxds%2BBWd5Tvoghom6hBZAESda6QWQQ8iPksDLvbT0w%3D%3D&X-Amz-Signature=17f7c1417798beb0c331327b5c7cfbd77e9cb8cbd2e174bcf39d635cf700edc3", status: 403, headers: {"x-amz-request-id": "H7RV6M7G8DH49RVJ", "x-amz-id-2": "tu5mR68nGEFM1uErsHI5V5o/3ANffjrXfuQSKygjkWDv+o44xfBHneRVZ/zIRoxK30xV7+ZKWXg=", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Fri, 10 Jul 2026 13:37:58 GMT", "connection": "close", "server": "AmazonS3"} }
test aws::tests::signed_url_conditional_create_blocks_overwrite ... FAILED
test aws::tests::signed_url_expires ... ok
failures:
failures:
aws::tests::signed_url_baseline_roundtrip_without_options
aws::tests::signed_url_conditional_create_blocks_overwrite
aws::tests::signed_url_multipart_multiple_parts_roundtrip
aws::tests::signed_url_multipart_upload
aws::tests::signed_url_query_value_with_space
aws::tests::signed_url_special_character_key
aws::tests::signed_url_tampering_is_rejected
aws::tests::signed_url_with_signed_checksum_header_is_enforced
aws::tests::signed_url_with_signed_content_type_is_enforced
test result: FAILED. 1 passed; 9 failed; 0 ignored; 0 measured; 174 filtered out; finished in 3.60s
error: test failed, to rerun pass `-p object_store --lib`
Which issue does this PR close?
Content-Typewhen signing object_store PUT requests #270Also related to #620 (versionId) and #318.
Rationale for this change
Right now
Signer::signed_urlcan only presign a bare URL. There's no way to add extra query parameters or to sign request headers, and S3 needs both.The big one is multipart uploads. To presign an
UploadPartrequest you have to includepartNumberanduploadIdas query parameters, and they have to be part of the signature. There's no way to do that today, so you can't hand out presigned URLs for the parts of a multipart upload.Signing headers matters for the same reason. If you want S3 to enforce a checksum, or pin the
Content-Type, or require SSE, those headers have to be baked into the signature when you presign. Otherwise the client can send whatever it wants.What changes are included in this PR?
SignedUrlOptionsand a newSigner::signed_url_optsmethod that takes extra query params and signed(name, value)header pairs along with the usual method/path/expiry.signed_url_optshas a default impl, and the existingsigned_urlnow just calls it with no options. So nothing changes for existing callers and Azure doesn't have to implement anything.+instead of%20, which doesn't match the canonical query, and the canonical query now sorts by the encoded(key, value)pair the way the spec wants.Are there any user-facing changes?
Yes, but only additions.
SignedUrlOptionsandsigned_url_optsare new.signed_urlbehaves exactly as before. No breaking changes — the trait method is defaulted, and the internal signing methods that got reworked were allpub(crate).This was tested end-to-end against real S3 (and MinIO): a full multipart round-trip with a sub-5MiB final part, checksum enforcement (accepts a matching body, rejects a tampered one, rejects a missing one),
Content-Typebinding, tampering withpartNumber/uploadId/signed headers all getting a 403, percent-encoded keys, expiry, andIf-None-Matchconditional create.