Question

After a routine driver upgrade on a DGX system with NVSwitch, several multi-GPU training jobs fail immediately at startup with NCCL errors, and nvidia-smi shows all GPUs present and healthy. Single-GPU jobs run fine. Checking the host, you find the fabric manager service is not running. What is the most likely cause and correct remediation?

Accepted Answer

The NVIDIA Fabric Manager service failed to start after the upgrade; restart/enable it so NVSwitch is initialized and NVLink peer-to-peer paths are established

Answer

The GPUs have overheated; drain the node and wait for thermal recovery before rescheduling jobs

Answer

MIG mode was enabled during the upgrade; disable MIG on all GPUs to restore multi-GPU communication

Answer

The InfiniBand adapters lost their subnet manager assignment; restart the subnet manager on the fabric