Question

A research team maintains a 2 PB collection of raw training datasets. Only a small subset is actively used at any time; most datasets are accessed a few times per year for occasional re-training experiments. The team wants to minimize storage cost for the rarely-used data while keeping the active working set on a high-throughput parallel filesystem for GPU training. Which storage approach best fits the rarely-accessed archive tier?

Accepted Answer

Place the rarely-accessed datasets on a cost-optimized object storage tier and stage subsets to the parallel filesystem when needed

Answer

Store the entire 2 PB on local NVMe SSDs inside each DGX node

Answer

Keep all 2 PB permanently on the high-performance parallel filesystem used for active training

Answer

Replicate the full 2 PB into each GPU's HBM memory for fastest access