Fix Xid healthcheck for replicated GPUs - #1992
Open
abraaoteixeira wants to merge 1 commit into
Open
Conversation
When a physical GPU experiences a critical Xid error, the device plugin needs to mark all associated logical devices (Time-Slicing, MPS) as unhealthy. Previously, the parentToDeviceMap was overwritten by replicated logical devices, causing only one of them to be marked unhealthy on an Xid event. This patch changes the mapping to map[string][]*Device so all logical devices bound to the failing physical UUID are properly marked. Fixes NVIDIA#1929 Signed-off-by: Abraão Teixeira <abraaoteixeira0101@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1929
This PR ensures that all logical devices (such as Time-Slicing or MPS replicas) sharing a physical GPU are properly marked as unhealthy when a critical Xid error occurs on the underlying hardware, instead of just the last registered replica.
(The
Signed-off-byis already included in the commit message).