Question

Your team serves a large image-classification model on a Vertex AI online endpoint backed by GPU replicas. Analytics show the endpoint receives near-zero traffic for 20 hours per day but must respond to a large volume of predictions submitted in a single burst each evening for a downstream reporting pipeline. Latency for individual predictions is not important; the reporting job only needs all results within a two-hour window. The current always-on GPU endpoint is expensive. What change best reduces cost while meeting the requirement?

Accepted Answer

Replace the online endpoint with a Vertex AI batch prediction job triggered each evening, letting Vertex provision and tear down compute for the job.

Answer

Keep the online endpoint but set minReplicaCount to 0 and rely on autoscaling to spin up GPU replicas when the evening burst arrives.

Answer

Deploy the model to an online endpoint with more GPU replicas and enable dynamic request batching to absorb the evening burst.

Answer

Distill the model to a smaller version and keep it on an always-on CPU online endpoint to lower per-hour cost.