Question

Your team hosts 12 lightweight image-classification models on Vertex AI, each receiving low but steady traffic (a few requests per second). Each model fits comfortably in about 2 GB of GPU memory. Currently each model runs on its own dedicated node with a full NVIDIA T4 GPU, and GPU utilization per node hovers around 8%. Leadership wants to cut serving costs without materially increasing latency. What is the most effective way to reduce cost here?

Accepted Answer

Deploy all 12 models to a single Vertex AI endpoint as co-hosted models sharing the same GPU-backed DeployedModel resources, consolidating them onto fewer GPU nodes

Answer

Switch every model to CPU-only nodes and enable dynamic batching to compensate for the loss of GPU acceleration

Answer

Keep one GPU per model but enable autoscaling with a minimum of zero replicas so idle models scale to nothing

Answer

Apply INT8 quantization to each model so they run faster and the GPUs finish requests sooner