Scaling prototypes into ML models

Your team trains an image classification CNN on a single NVIDIA A100 GPU using a per-device batch size of 64 and a base learning rate that converges well. To reduce wall-clock time, you scale to 8 A100 GPUs on one node using data-parallel synchronous training with tf.distribute.MirroredStrategy, keeping the same per-device batch size of 64. After the change, training loss decreases much more slowly per epoch and final accuracy drops compared to the single-GPU run. What is the most appropriate first adjustment?

Your team trains a large PyTorch image model on Vertex AI custom training. Each run takes about 30 hours on a single node with multiple GPUs. To reduce cost, you switched to Spot (preemptible) VMs, but you notice that when a VM is reclaimed, the job restarts from scratch and wastes many GPU-hours. You must keep costs low while ensuring interrupted runs can resume efficiently. What should you do?

Your team is training a large image classification CNN on a single NVIDIA A100 GPU using TensorFlow. To fit a larger batch size and speed up training, you enabled mixed-precision (float16) training. After the change, throughput improved significantly, but the loss frequently becomes NaN early in training and the model fails to converge. What is the most appropriate action to fix the instability while keeping the performance benefits of mixed precision?

Your team has prototyped a gradient-boosted decision tree model (XGBoost) on a small tabular dataset for predicting equipment maintenance needs. You now need to scale training to a 200 GB dataset and later serve batch predictions nightly. A junior engineer suggests provisioning a cluster of GPU-accelerated VMs for both training and inference to maximize speed. What is the most cost-effective and appropriate hardware recommendation for this workload?

A data science team at a biotech firm has a labeled dataset of only 1,800 records for a tabular classification task predicting patient treatment response. They are prototyping several model architectures and want a reliable estimate of generalization performance before committing to a full training and hyperparameter tuning run on Vertex AI. A single hold-out validation split gave them wildly different accuracy numbers each time they reshuffled the data. What is the most appropriate approach to obtain a robust performance estimate given the dataset size?

Your team is training a 40-billion-parameter transformer language model from scratch. Each model replica's parameters, optimizer states, and activations far exceed the 40 GB of memory available on a single GPU, so the full model cannot fit on one device. You have access to a multi-node cluster of GPUs on Vertex AI custom training. Which distributed training approach should you adopt to make training feasible?

Your team has a TensorFlow model that trains on a 4 TB dataset stored in Cloud Storage as TFRecord files. A single-node 8-GPU machine takes over 40 hours per training run, which is blocking your experimentation velocity. The model uses large embedding layers and dense transformer blocks, and the training loop is written with the Keras API. You need to dramatically reduce wall-clock training time while keeping code changes minimal and controlling cost. What is the most appropriate approach?

Your team has a TensorFlow 2 image classification model that currently trains on a single GPU and takes 18 hours per run. You provisioned a Vertex AI custom training job on one machine equipped with 4 NVIDIA A100 GPUs and want to reduce training time with minimal code changes while keeping the model architecture unchanged. Which approach should you use to distribute training across the 4 GPUs?

Your team has a prototype trained in scikit-learn on a small sample of tabular data. The production dataset has 4 billion rows stored in BigQuery, with several high-cardinality categorical features. You must train a gradient-boosted model that scales to the full dataset, supports distributed training across multiple workers, and integrates cleanly with Vertex AI managed training. Which approach best meets these requirements?

You are training a large image classification CNN on a single Vertex AI custom training node equipped with one NVIDIA A100 (40 GB) GPU. Training crashes with a CUDA out-of-memory error only after you increased the input image resolution from 224x224 to 512x512. The model architecture and dataset are fixed, and you must keep the higher resolution. You want the simplest change that lets training proceed on the same single-GPU node without significantly degrading final model accuracy. Which action should you take first?

Your team is training a large vision transformer on a single NVIDIA A100 (40 GB) GPU using PyTorch on Vertex AI. Research shows the model converges best with an effective batch size of 512, but the GPU runs out of memory at a batch size of 64. You must not change hardware and want to preserve the target effective batch size without degrading convergence. What is the most appropriate technique?

Your team is running Vertex AI hyperparameter tuning on a deep learning model that takes about 6 hours per trial. You have a fixed budget of 40 total trials and want the best possible objective value. You are using Bayesian optimization as the search algorithm. Your manager wants results delivered as quickly as possible, but also wants the search to be as effective as possible. How should you configure the maxParallelTrials setting to balance wall-clock time and search quality?

You are training a TensorFlow image classification model on a single Vertex AI custom job with one NVIDIA A100 GPU. Training is slow, and monitoring shows GPU utilization averaging only 25%, with frequent idle periods. The dataset consists of millions of JPEG files stored in Cloud Storage and is read one file at a time inside a Python loop before being converted to tensors. What is the most effective change to improve training throughput?

Your team is training a deep neural network for image classification on Vertex AI using 8 GPUs with data parallelism. To improve throughput, you increased the global batch size from 256 to 4096 and proportionally scaled the base learning rate. However, training now diverges within the first few hundred steps, with the loss quickly becoming NaN. The single-GPU baseline with batch size 256 trained stably. What is the most appropriate change to stabilize training while retaining the large batch size?

Your team trains a moderately sized image classification model (ResNet-50) on a single Vertex AI custom training job with one NVIDIA A100 GPU. Training is slower than expected, and monitoring shows GPU utilization averaging only 25% with frequent dips to near zero, while CPU utilization is pegged near 100%. Before scaling to more GPUs, what is the most appropriate first action to improve training throughput?

Your team trains a large PyTorch image model on Vertex AI using data-parallel synchronous SGD across multiple nodes, each with 8 NVIDIA A100 GPUs. Training is bottlenecked by the all-reduce gradient synchronization step, and network communication between worker nodes dominates step time. You need to reduce this communication overhead and improve overall training throughput without changing your model code significantly. What should you do?

Your team trains a TensorFlow image classification model on a Vertex AI single-GPU instance. The dataset is 200 GB of TFRecords stored in Cloud Storage and does not fit in memory. During profiling you notice the GPU sits idle between steps, and each epoch reads the data from Cloud Storage anew. You want to maximize GPU utilization and reduce redundant reads without changing hardware. Which tf.data pipeline configuration should you apply?

A data science team has prototyped a deep neural network in TensorFlow/Keras for a tabular churn prediction problem. Training on a single n1-standard-16 CPU instance takes 14 hours per run, which is blocking their experimentation. The model uses dense layers with large matrix multiplications, a batch size of 4096, and a dataset of 20 million rows stored as sharded TFRecords in Cloud Storage. Profiling shows the CPU is compute-bound (near 100% utilization) during the training step, not I/O bound. What is the most effective change to reduce per-run training time while minimizing code changes?

Your team has a working prototype: a gradient-boosted tree model trained in a scikit-learn notebook on a 5 GB CSV file. Product wants this retrained weekly on a growing dataset (now 400 GB, stored in Cloud Storage as sharded Parquet files) with minimal custom infrastructure. The data scientists prefer to keep writing Python and want managed, repeatable training runs. Which approach best scales this prototype for production?

Your team has a research prototype written in PyTorch that trains a custom convolutional network on a single GPU. Leadership wants to productionize it on Vertex AI with minimal rewrite, full support for distributed multi-worker training, and native model serving through Vertex AI Prediction with GPU acceleration. The lead researcher is comfortable keeping the code in PyTorch. What is the most appropriate approach for scaling this prototype into a production training and serving pipeline?