Question

A logistics company is designing an agent embedded in its warehouse mobile app that classifies inbound shipment notes into predefined categories. The task uses a narrow, well-defined vocabulary, must respond in under 200 milliseconds, and will process millions of requests per day at high concurrency. The architecture team wants to minimize inference cost while meeting the latency target. Which approach best fits these requirements?

Accepted Answer

Deploy a fine-tuned small language model (SLM) optimized for the classification task

Answer

Use a frontier large language model (LLM) with a detailed system prompt for each request

Answer

Route every request through Microsoft 365 Copilot to leverage organizational grounding

Answer

Build a custom foundation model from scratch trained on the company's shipment data