Question

Your team serves a product recommendation model on a Vertex AI online endpoint. Each request requires computing dense embeddings for a fixed catalog of 2 million products, then scoring the user against those embeddings. The catalog changes only once per day. Currently, product embeddings are recomputed on every request, causing p95 latency to exceed 800 ms and driving up GPU costs. You must reduce online latency while keeping recommendations fresh within one day. What is the best approach?

Accepted Answer

Precompute all product embeddings in a nightly batch prediction job, store them in a low-latency store (e.g., Vertex Feature Store or a vector index), and have the online endpoint only compute the user embedding and perform the lookup/scoring at request time.

Answer

Increase the online endpoint's minimum replica count and attach larger GPUs so the per-request embedding computation completes faster.

Answer

Enable dynamic request batching on the online endpoint so multiple user requests share the product-embedding computation.

Answer

Apply post-training int8 quantization to the embedding model so each per-request embedding computation runs faster on the same hardware.