Question

A research team is deploying a large language model whose weights are too large to fit within the memory of a single GPU. They want to run the model across four GPUs in one server, splitting the model's layers and parameters across the devices so that each GPU holds only a portion of the network. Which technique best describes this approach?

Accepted Answer

Model parallelism, where the model itself is partitioned across multiple GPUs

Answer

Data parallelism, where each GPU holds a full copy of the model and processes different batches of input

Answer

Gradient checkpointing, where activations are recomputed to reduce memory use

Answer

Mixed-precision training, where lower-precision formats reduce memory footprint