Question

A data scientist notices that a deep learning training job is running far slower than expected on a GPU server. Profiling reveals that during every training step, the GPU sits mostly idle while large volumes of data are repeatedly copied from system (CPU) memory across the PCIe bus to the GPU. Which fundamental GPU computing principle explains why this pattern degrades performance, and what should be optimized?

Accepted Answer

GPUs achieve peak throughput only when data resides in high-bandwidth on-device memory, so frequent CPU-to-GPU transfers over the slower PCIe bus create a bottleneck; keeping data resident in GPU memory and overlapping transfers reduces the stall.

Answer

GPUs cannot process data stored in system RAM at all, so the job must be moved entirely to the CPU to run correctly.

Answer

The PCIe bus is faster than GPU memory, so the fix is to force all data through system RAM on every step rather than caching it on the GPU.

Answer

GPU cores automatically prefetch all data from disk, so the slowdown must be caused by an undersized training batch rather than any data movement issue.