Question

You have trained a 40 GB transformer model that must be served for online inference with sub-200ms latency. The largest single GPU available in your region has 24 GB of memory, so the model does not fit on one accelerator. You must deploy it to a Vertex AI Endpoint while keeping inference latency low. Which approach best enables serving this model?

Accepted Answer

Split the model across multiple GPUs on a single machine using tensor/pipeline parallelism in the serving container, deploying to a multi-GPU machine type

Answer

Convert the model to a batch prediction job so the whole model can be loaded into system RAM instead of GPU memory

Answer

Increase the endpoint's minimum replica count so the model is distributed across replicas automatically

Answer

Enable dynamic batching on the endpoint to reduce the per-request memory footprint enough to fit on one 24 GB GPU