fix: reuse a shared awscrt ClientBootstrap to stop event loop thread leak - #75
Merged
Merged
Conversation
…leak get_rest_devices built its own EventLoopGroup, DefaultHostResolver and ClientBootstrap on every call. awscrt starts a native event loop thread per EventLoopGroup and only stops it when the native resource is destroyed, which never happens here: the per-device shadow subscriptions hold callbacks that reference the device, which references the connection, which owns the bootstrap, so the graph stays reachable even after disconnect(). Callers rebuild the connection whenever the AWS credentials expire, and Hatch issues those hourly, so every refresh stranded another thread along with its pipe pair and CRT buffers. On a long-running Home Assistant instance this showed up as ~24 leaked AwsEventLoop threads per day and steady RSS growth; one install reached 213 leaked threads over 8 days of uptime. Use awscrt's process-wide static default bootstrap instead, which keeps the cost flat at a single event loop thread no matter how often we reconnect. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Aug 25, 2026
Closed
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
get_rest_devices()built a freshio.EventLoopGroup(1)+DefaultHostResolver+ClientBootstrapon every call. awscrt starts a native thread perEventLoopGroupand only stops it when the native resource is destroyed. Nothing in the shutdown
path ever destroys it: each device's shadow subscriptions hold callbacks that
reference the device, which references the connection, which owns the bootstrap —
so the whole graph stays reachable even after
disconnect().Home Assistant's
ha_hatchintegration rebuilds this connection every time theAWS credentials expire, which Hatch issues hourly, so a long-running instance
strands one
EventLoopGroupthread per hour indefinitely. Measured on my owninstance: 215 leaked
AwsEventLoopthreads after 8.7 days of uptime, an exactmatch for the reconnect count, costing roughly 80 MB/day.
Fix: use
io.ClientBootstrap.get_or_create_static_default(), awscrt's ownprocess-wide default, instead of allocating a new one per call.
Test plan
EventLoopGroup/DefaultHostResolverare built per call, and a secondconnection reuses the same bootstrap instance. All fail against the
pre-fix code (
AttributeError/mock-call-count mismatches), confirmingthey aren't vacuous.
git+httpsrequirement pointed atthis branch and soaked in production.
AwsEventLoopthread count held flatat 2 across 137 hours of uptime (vs. 215 pre-fix over 8.7 days), and open fds
stayed flat at 72-77 (vs. 434 pre-fix).
There is a second, independent leak in the same area — shadow subscriptions are
never unsubscribed, so the Python-side connection graph also outlives
disconnect(). That's a separate PR on top of this branch, since the two areindependently reviewable and this one already has its own soak evidence.