Question

Your team has deployed 12 small TensorFlow models to Vertex AI, each on its own dedicated online endpoint with one n1-standard-4 node. Traffic to each model is low and sporadic, but the endpoints must remain available for occasional real-time requests. Finance flags that you are paying for 12 always-on nodes even though average utilization per endpoint is under 5%. You need to significantly reduce serving cost while keeping low-latency online inference available. What should you do?

Accepted Answer

Deploy multiple models to a single Vertex AI endpoint and place them on a shared DeployedModel/node pool via co-hosting (model co-hosting / deployment resource pools), sharing compute across the low-traffic models.

Answer

Switch all 12 models to Vertex AI batch prediction jobs and run them hourly so no nodes stay running between requests.

Answer

Keep the 12 endpoints but reduce each machine type to n1-standard-2 and disable autoscaling to lock in a lower fixed cost.

Answer

Merge the 12 models into one large ensemble model artifact and deploy it to a single endpoint with a GPU accelerator.