Question

A research team runs large-scale distributed training across 32 DGX nodes connected with NVIDIA Quantum-2 InfiniBand. Profiling shows that the all-reduce collective operations during gradient synchronization consume a growing share of each training step as they scale out. The infrastructure architect wants to reduce this communication overhead by offloading part of the reduction work from the GPUs and host CPUs. Which InfiniBand capability directly addresses this?

Accepted Answer

SHARP (Scalable Hierarchical Aggregation and Reduction Protocol), which performs reduction operations inside the network switches

Answer

Adaptive routing, which spreads packets across multiple paths to avoid congestion hotspots

Answer

Increasing the MTU size on the InfiniBand host channel adapters to reduce packet overhead

Answer

GPUDirect RDMA, which allows the NIC to read and write GPU memory without CPU involvement