Question

Your team is training a deep learning model on a single Vertex AI GPU VM using TensorFlow. The full training dataset is 800 GB of image files stored in Cloud Storage, far larger than the VM's RAM or GPU memory. Early attempts crash because the training script tries to load the entire dataset into memory before training. You need the most efficient way to feed this dataset to the model without running out of memory. What should you do?

Accepted Answer

Convert the dataset to sharded TFRecord files and build a tf.data pipeline that streams and prefetches batches directly from Cloud Storage during training.

Answer

Provision a VM with 1 TB of RAM so the entire dataset can be loaded into memory once before training begins.

Answer

Randomly downsample the dataset to a size that fits in GPU memory and train only on that subset.

Answer

Load the full dataset into a single pandas DataFrame and pass it to model.fit() with a large batch size.