Question

Your team is training a large convolutional neural network in TensorFlow on Cloud TPU v3 Pods. The model trained fine on GPUs using float32, but your ML engineer wants to maximize TPU throughput. They ask which numeric format the TPU matrix units natively use and how to adapt the code for best performance while preserving numerical stability. What should you advise?

Accepted Answer

TPUs natively compute matrix multiplications in bfloat16; keep variables and reductions in float32 and let the compiler cast matmul inputs to bfloat16, which preserves dynamic range without requiring manual loss scaling.

Answer

TPUs require float16 (IEEE half-precision); you must manually convert all weights and activations to float16 and add dynamic loss scaling to avoid gradient underflow.

Answer

TPUs only support float32; there is no lower-precision path, so no code changes can improve throughput and you should switch to GPUs for mixed precision.

Answer

TPUs use int8 quantization for training; you must quantize the model with a post-training quantization tool before submitting the TPU training job.