Question

A team is deploying a single 8-GPU server to train a large language model that uses tensor parallelism, requiring very frequent all-to-all exchanges of activation tensors between all GPUs in every forward and backward pass. During evaluation, they notice that on a PCIe-only server the GPUs spend significant time waiting on inter-GPU communication, and scaling efficiency drops sharply as they add GPUs. Which server interconnect design would BEST relieve this communication bottleneck within the node?

Accepted Answer

An HGX baseboard where all 8 GPUs are fully connected through NVSwitch, providing high-bandwidth non-blocking GPU-to-GPU communication

Answer

A PCIe Gen5 server with additional PCIe switches to increase the number of available lanes per GPU

Answer

A server that routes all inter-GPU traffic through the host CPU's system memory over separate PCIe root complexes

Answer

A server that adds more InfiniBand NICs so intra-node GPU traffic can traverse the external network fabric