Question

Your team trained a small convolutional network (about 4 million parameters) on a single-host TPU v3-8. During training you observe that step time is dominated by host-to-device data transfer and the TPU MXU utilization reported in the profiler is consistently below 10%. The model fits comfortably in memory and the dataset is only 12 GB. Management asks you to reduce training cost without significantly changing the model architecture. What is the most appropriate action?

Accepted Answer

Switch to a single GPU (for example an NVIDIA T4 or L4) instance, since the small model cannot saturate the TPU's matrix units and a GPU offers better cost-efficiency for this workload

Answer

Scale up to a TPU v3-32 Pod slice so more cores share the workload and drive utilization higher

Answer

Keep the TPU but reduce the batch size to lower memory pressure and improve utilization

Answer

Convert the model to use bfloat16 mixed precision to better exploit the TPU MXU