Question

A data center team is deploying a cluster of DGX H100 systems for large-scale distributed training. During network fabric planning, they want to ensure that high-volume dataset reads from the shared parallel file system do not compete with the GPU-to-GPU gradient exchange traffic during collective operations. What is the recommended approach on the DGX H100 platform to achieve this separation?

Accepted Answer

Use dedicated InfiniBand adapters for the compute (east-west) fabric and separate network adapters for the storage fabric

Answer

Route both storage and compute traffic over the same set of ConnectX-7 adapters and rely on QoS priority queues to arbitrate

Answer

Send all gradient exchange traffic over NVLink and all storage traffic over a single 10GbE management port

Answer

Disable GPUDirect RDMA so storage and compute traffic are serialized through the CPU, avoiding contention