Serving and scaling models

Your team serves a product recommendation model on a Vertex AI online endpoint. Each request requires computing dense embeddings for a fixed catalog of 2 million products, then scoring the user against those embeddings. The catalog changes only once per day. Currently, product embeddings are recomputed on every request, causing p95 latency to exceed 800 ms and driving up GPU costs. You must reduce online latency while keeping recommendations fresh within one day. What is the best approach?

A retail analytics team needs to score 400 million customer records once per week using a trained TensorFlow model deployed on Vertex AI. The records live in a large BigQuery table. Latency is not a concern, but the team wants the job to complete as quickly and cheaply as possible while avoiding the operational overhead of managing infrastructure. Which approach best meets these requirements?

A retail analytics team needs to score their entire customer base (about 40 million records) once every night to generate propensity scores for a next-day marketing campaign. The scores are consumed by a downstream BigQuery table the following morning, so end-to-end latency of a few hours is acceptable. The team currently keeps a Vertex AI online endpoint with multiple GPU replicas running 24/7 to handle this workload, but finance has flagged the cost as excessive. What is the most appropriate change to reduce cost while meeting requirements?

Your team maintains a fraud-scoring model deployed on a Vertex AI online endpoint. A new business requirement asks you to re-score the entire historical transaction table (2 billion rows) once, to backfill risk labels for an analytics dashboard. There is no latency requirement for this one-time job, but you want to minimize cost and total wall-clock time. The online endpoint currently autoscales between 3 and 10 GPU replicas. What is the most appropriate approach?

A retail company runs a 340M-parameter BERT-based text classifier on a Vertex AI endpoint to categorize incoming customer chat messages in real time. The model meets accuracy targets, but p99 latency is 480 ms — well above the 120 ms SLA — and GPU serving costs are high. The team wants to dramatically reduce latency and cost while keeping accuracy within about 2% of the current model, and they are willing to invest in a one-time offline effort. Which approach best meets these requirements?

Your team serves a large BERT-based text classifier on a Vertex AI endpoint with GPU accelerators. Traffic has grown, and GPU serving costs are becoming unsustainable. The business requires that classification accuracy remain within 1% of the current model, and that p95 latency stay under 50 ms. You want to reduce serving cost the most while meeting both constraints, and you have access to the full labeled training dataset plus abundant unlabeled production text. Which optimization approach should you pursue first?

You run a TensorFlow image-classification model on a Vertex AI online endpoint backed by GPUs. Traffic is bursty, and during peak periods GPU utilization stays low (around 30%) even though request latency climbs and clients report timeouts. Each individual request contains a single image. You want to increase throughput and improve GPU efficiency without reducing model accuracy or provisioning more replicas. What is the most effective change?

Your team hosts 12 lightweight image-classification models on Vertex AI, each receiving low but steady traffic (a few requests per second). Each model fits comfortably in about 2 GB of GPU memory. Currently each model runs on its own dedicated node with a full NVIDIA T4 GPU, and GPU utilization per node hovers around 8%. Leadership wants to cut serving costs without materially increasing latency. What is the most effective way to reduce cost here?

Your team serves a TensorFlow image-classification model on a Vertex AI endpoint. A high-volume internal microservice sends millions of prediction requests per hour and is experiencing significant per-request serialization overhead and connection setup latency. The payloads are large tensor arrays. You want to reduce serving latency and increase throughput between the client and the model server without changing the model itself. Which change should you prioritize?

A financial services company serves a large transformer-based text classification model on Vertex AI endpoints backed by NVIDIA T4 GPUs. Traffic has tripled, and p99 latency now regularly exceeds the 150 ms SLA. The team wants to increase throughput per GPU without adding more replicas or retraining the model, and they can tolerate a very small drop in accuracy. Which optimization should they apply first?

Your team has deployed 12 small TensorFlow models to Vertex AI, each on its own dedicated online endpoint with one n1-standard-4 node. Traffic to each model is low and sporadic, but the endpoints must remain available for occasional real-time requests. Finance flags that you are paying for 12 always-on nodes even though average utilization per endpoint is under 5%. You need to significantly reduce serving cost while keeping low-latency online inference available. What should you do?

Your team trained several models in different frameworks (a TensorFlow SavedModel, a PyTorch model exported to TorchScript, and an ONNX classifier). You want to serve all of them behind a single high-performance inference server on a Vertex AI endpoint with GPUs, minimizing custom container maintenance while supporting multiple frameworks and features like concurrent model execution. Which approach best meets these needs?

Your team serves a fraud-scoring model on a Vertex AI online endpoint. Median (p50) latency is well within your 40 ms SLO, but the p99 latency regularly spikes to 300 ms, causing occasional timeouts for the payment service. CPU utilization is moderate and autoscaling is not triggering. Investigation shows the spikes correlate with garbage collection pauses and large individual request payloads that occasionally arrive together. Which action most directly reduces the p99 tail latency without over-provisioning?

You deployed a large PyTorch recommendation model to a Vertex AI endpoint with autoscaling configured to scale down to zero replicas during off-peak hours to save cost. Users report that the first requests after idle periods experience 15+ second latencies, while steady-state requests return in under 200 ms. Your SLA requires p99 latency below 1 second at all times. What is the most effective change to meet the SLA?

Your company runs a fraud-scoring model on a Vertex AI endpoint. A downstream microservice inside your VPC calls the model synchronously for every transaction, and you must minimize both network latency and exposure of the prediction traffic to the public internet. Measurements show that the current public endpoint adds unacceptable round-trip latency because traffic egresses and re-enters Google's network. Which change best reduces latency while keeping traffic off the public internet?

Your team deployed an INT8 post-training quantized version of a sentiment classification model to reduce serving latency and cost. After deployment, accuracy dropped from 94% to 87%, which is below the acceptable 92% threshold, while the FP32 model met the threshold. You want to keep the smaller INT8 footprint and low latency benefits but recover most of the lost accuracy. What is the most effective next step?

You deploy a TensorFlow model on a Vertex AI endpoint using a custom serving container that runs a single inference server process per replica. Each replica is provisioned with 8 vCPUs and no GPU. Under load tests, individual requests complete in about 40 ms, but overall endpoint throughput plateaus far below what the CPU capacity suggests, and CPU utilization stays around 25%. Latency remains flat until the plateau, then rises sharply. You want to raise throughput per replica before adding more replicas. What is the most effective first step?

Your team has trained a churn-prediction model registered in Vertex AI. Every week, the marketing analytics team needs scores for the entire customer base (about 40 million rows currently stored in a BigQuery table) so they can join the results with other campaign tables in BigQuery for reporting. There is no requirement for real-time responses, and the job must run cheaply and fully unattended. Which serving approach best fits these requirements?

Your team serves a large vision transformer model on a Vertex AI online endpoint. The model receives sustained, high-volume traffic during business hours, and each request runs a compute-intensive forward pass. Current single-GPU replicas hit their throughput ceiling and requests queue, driving up latency, even though CPU and memory usage remain low. You need to increase per-replica throughput while keeping per-inference cost reasonable. What is the most effective first step?

You deployed a computer vision model to a Vertex AI online prediction endpoint backed by GPU-equipped nodes. Traffic is bursty: during business hours request volume spikes 10x, but at night it drops to near zero. During spikes, p99 latency exceeds your 200 ms SLO, while at night you are paying for idle GPU nodes. You want to keep latency within SLO during spikes while minimizing cost during quiet periods, with minimal ongoing management. What should you do?