Skip to content

Shutdown internally-created executor on error in TransferManager - #3427

Merged
bhoradc merged 2 commits into
version-3from
fix/gh-3419-transfer-manager-executor-shutdown
Sep 24, 2026
Merged

bhoradc merged 2 commits into
version-3from
fix/gh-3419-transfer-manager-executor-shutdown

Conversation

@bhoradc

@bhoradc bhoradc commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #3419

Description

TransferManager shuts down its internally-created DefaultExecutor after the operation completes. Because the shutdown call is not in an ensure block, any error raised during the transfer skips it, leaving the executor's worker threads parked on its internal queue forever. In long-running processes with repeated failures this accumulates until
thread or memory limits are hit.

Present in all five methods that create an internal executor:
upload_file, upload_stream, download_file, upload_directory, download_directory.

Fix

Wrap each operation body in begin/ensure so executor.shutdown unless @executor runs on both success and error paths. The unless @executor guard is preserved, so custom executors remain the caller's
responsibility. DefaultExecutor#shutdown is idempotent, so no double-shutdown risk.

Testing

Added one regression spec per method. Each stubs a failure, captures the internally-created DefaultExecutor, and asserts shutdown was called on it. All five fail on main and pass with this change.

Locally ran the full CI-equivalent suite (rake build, rake test:spec, rbs collection install && rake rbs:test) - all green.

AI-assisted and manually reviewed.


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@bhoradc
bhoradc marked this pull request as ready for review September 24, 2026 00:33
@bhoradc
bhoradc requested a review from a team as a code owner September 24, 2026 00:33
@github-actions

Copy link
Copy Markdown

Detected 1 possible performance regressions:

  • aws-sdk-cloudwatchlogs.put_log_events_small_allocated_kb - z-score regression: 58.68 -> 58.82. Z-score: Infinity

@jterapin jterapin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once you address the below comment regarding object customization, we should be good to approve.

Unrelated: I found some bugs relating to TM so I will have a fix out shortly.

Comment thread gems/aws-sdk-s3/lib/aws-sdk-s3/transfer_manager.rb
@bhoradc
bhoradc requested a review from jterapin September 24, 2026 18:25

@jterapin jterapin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@bhoradc
bhoradc merged commit 6b4aae6 into version-3 Sep 24, 2026
34 of 35 checks passed
@bhoradc
bhoradc deleted the fix/gh-3419-transfer-manager-executor-shutdown branch September 24, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Thread leak on error in Aws::S3::TransferManager

3 participants