Repository navigation
fix(events): GCS rate limits no longer fail the webhook - #1367
Draft
aivong-openhands wants to merge 4 commits into
Draft
aivong-openhands wants to merge 4 commits into
aivong-openhands wants to merge 4 commits into
Conversation
…lerate rate limits A busy conversation renamed index.json for every saved event, past the per-object mutation limit of GCS. The 429 failed the webhook, and the agent-server dropped events when its queue filled. Fixes #1344
A rate-limited invalidation left index.json without the batch's events until another event arrived. Retry it off the request path with exponential backoff and jitter, one retry per conversation, and skip invalidations while it is pending.
This comment has been minimized.
This comment has been minimized.
Co-authored-by: openhands <openhands@all-hands.dev>
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||||||||
Runs the GCS event service against fake-gcs-server, with a session that answers 429 to a second change to one object within a short window, as GCS does. Every saved event must still be found once the retries finish.
aivong-openhands
changed the title
aivong-openhands
changed the title
aivong-openhands
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HUMAN:
AGENT:
Why
Each event that reached the event webhook renamed the conversation's
index.jsontoindex_stale.json. A busy conversation did this several times a second. Google Cloud Storage (GCS) allows about one change per second to one object, so it returned 429. The 429 failed the webhook. The agent-server then dropped events when its queue filled.Summary
SlowDownfrom S3, during invalidation no longer fails the webhook. The app retries the invalidation in the background, with exponential backoff and jitter for up to 30 seconds. While that retry is pending, saves for the same conversation skip invalidation, because the retry covers their events.Issue Number
Fixes #1344
How to Test
tests/unit/app_server/test_filesystem_event_service.pychecks that saving three events invalidates the index once. It also checks that a rate-limited invalidation is retried in the background, and that saves skip invalidation while the retry is pending.test_google_cloud_event_service.pyandtest_aws_event_service.pycheck that each store reports a rate limit.test_google_cloud_event_service_rate_limit.pyruns the GCS event service against fake-gcs-server in Docker. Its HTTP session answers 429 to a second change to one object within 0.2 seconds, as GCS does within one second. The test saves 10 batches while searching, and then finds every saved event.Video/Screenshots
Rate-limited event storage integration test: Ten batches save 30 events while searches run. An injected rate limit triggers a background retry. The final search finds every saved event, with no missing or unexpected IDs. The test uses a local GCS emulator with a 0.2-second rate-limit window. It does not test production GCS, S3, or HTTP webhook responses.
rate-limited-event-storage-1367.mp4
Type
Notes
While a retry is pending,
index.jsonis missing the newest events, so a search can miss them for up to 30 seconds. If the retry still fails after 30 seconds, the app logs an error, and the index stays stale until the next save to that conversation. The retry state is per app process. A restart drops a pending retry, and several replicas can each rename the same index.A search rebuilds
index.json, and the next batch renames it again. So an open conversation page can still change the index more than once a second.This PR was drafted by an AI agent on behalf of the user.
Enterprise server image for this PR: