Add random wait time to aborted join retry - #63
Open
niecore wants to merge 1 commit into
Open
Conversation
Avoid a deadlock when multiple nodes request the global lock at the same time: global:trans/4 is called with zero retries, so when several nodes each grab the lock on part of the cluster, the transaction is aborted on all of them and every node retries immediately in lockstep, colliding again and again. Sleeping for a random 1-3000 ms before retrying desynchronizes the nodes so that one of them can acquire the lock on all nodes and the joins proceed one after another. Verified with cets_join_SUITE (join_retried_if_lock_is_busy covers the aborted-retry path); results match the parent commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Avoid a deadlock when multiple nodes request the global lock at the same time: global:trans/4 is called with zero retries, so when several nodes each grab the lock on part of the cluster, the transaction is aborted on all of them and every node retries immediately in lockstep, colliding again and again.
Sleeping for a random 1-3000 ms before retrying desynchronizes the nodes so that one of them can acquire the lock on all nodes and the joins proceed one after another.
This issue was observed by running @mongoose-im in CETS cluster and adding the 7th or 8th node:
When the new node connected, the CETS discovery process on every node simultaneously tried to join each of replicated tables with it. Every join must first acquire a single cluster-wide lock via global:trans — with Retries = 0 (one shot: grants needed from all nodes, any collision aborts) and an immediate retry loop with no backoff in cets_join:join_loop.