Question

A retail company deploys a recommendation model as a real-time API. Users expect responses in under 100 milliseconds. The MLOps team notices that increasing the inference batch size improves the number of requests processed per second but causes some individual requests to exceed the response time target. Which characteristic of inference workloads best explains this observation?

Accepted Answer

Larger batch sizes increase throughput but can add latency for individual requests because requests must wait to be grouped together before processing.

Answer

Larger batch sizes always reduce both latency and throughput because the GPU processes fewer operations overall.

Answer

Inference workloads require backpropagation, so batching triggers gradient updates that slow individual responses.

Answer

Latency and throughput are identical metrics, so improving one automatically degrades the model's accuracy.