Repository navigation
fix: raise rccl_send_recv ctest timeout to 300s - #562
Merged
zichguan-amd merged 1 commit intoOct 27, 2026
Merged
Conversation
The 60s TIMEOUT was calibrated on 8 GPUs. On a 64-GPU MI350X CPX node the sample needs about as long as the other RCCL examples and is killed before ncclCommInitAll finishes. QA confirmed 300s completes. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
rccl_send_recvis the only RCCL example withTIMEOUT 60, added in #558 from 8-GPU timings (~15 s on MI210). On a 64-GPU MI350X CPX node (ROCM-33484) CTest kills it at ~61 s, after the ring topology print and beforencclCommInitAllreturns. The other RCCL samples in that same run have no per-test timeout and pass in 180–218 s. QA re-ranrccl_send_recvwith a 300 s timeout and it completed.Technical Details
Libraries/RCCL/send_recv/CMakeLists.txt:TIMEOUT 60→TIMEOUT 300.Test Plan
Libraries/RCCL/send_recvagainst the host ROCm on an idle Ruby MI350 node and timerccl_send_recvwith all visible GPUs.Test Result
Ruby
cv350-rck-g03-c07-18(8× MI350X, gfx950, SPX/NPS1, ROCm 7.0.2.2):rccl_send_recvpassed in 39.9 s. That is under 60 s, so this node does not reproduce the failure. Accelerator partition profiles are reported as N/A, so the node cannot be switched to the 64-GPU CPX/NPS2 layout from the ticket. 39.9 s on 8 GPUs also shows the 60 s cap has little headroom even before 64 ranks. The 300 s value is the one QA confirmed on the 64-GPU host.Added/Updated documentation?
Included Visual Studio files?
Submission Checklist
Made with Cursor