Question

Your team is training a TensorFlow image classification model on Vertex AI custom training with 4 GPUs. The dataset consists of 8 million JPEG images stored as thousands of individual files in Cloud Storage. During training, GPU utilization stays around 25%, and profiling shows the accelerators are frequently idle waiting for data. You must improve training throughput without changing the model architecture. What is the most effective change to the data storage and pipeline?

Accepted Answer

Convert the images into a set of sharded TFRecord files and read them with tf.data using interleave, parallel map, and prefetch.

Answer

Move all images into a single large TFRecord file so the pipeline only opens one file during training.

Answer

Increase the number of GPUs from 4 to 8 to process the incoming data faster.

Answer

Load the entire dataset into a pandas DataFrame in memory before starting training.