Question

An AI infrastructure team is deploying a multi-rack GPU cluster for large-model training. They need shared, high-throughput access to a multi-petabyte dataset from every compute node while ensuring dataset reads never compete for bandwidth with the GPU-to-GPU gradient synchronization traffic. Which infrastructure design best meets this requirement?

Accepted Answer

Provision a dedicated high-speed storage fabric with a parallel file system, kept physically separate from the compute (east-west) fabric used for inter-GPU communication

Answer

Attach local NVMe drives to each node and copy the full dataset to every server before each training run

Answer

Route both storage traffic and inter-GPU collective traffic over the same InfiniBand compute fabric to simplify cabling

Answer

Store the dataset in a single node's memory and stream it to all other nodes over the management Ethernet network