Question

Your team deployed a fine-tuned language model to an Azure Machine Learning managed online endpoint that is integrated with your Foundry project. During business hours, traffic is highly variable, with sudden spikes that occasionally cause request queuing and timeouts, while overnight traffic is nearly zero. You must minimize cost during idle periods while automatically handling spikes without manual intervention. What should you configure?

Accepted Answer

Attach an autoscale rule based on CPU/GPU and request-latency metrics that scales the deployment's instance count up and down within min/max bounds.

Answer

Set a fixed instance count sized for peak traffic on the deployment so capacity is always available.

Answer

Deploy the model to a serverless API endpoint and rely on the model provider's default rate limits.

Answer

Increase the request timeout on the endpoint and enable request retries in the client SDK.