Question

A team has deployed a customer-support RAG agent to production in Azure AI Foundry. They already stream production interactions to a sampled online evaluation that computes groundedness and relevance on a percentage of live traffic. Management now wants to be automatically notified when the agent's groundedness score deteriorates in production so engineers can investigate before customers are widely affected. Without building any custom code, what should the team configure to meet this requirement?

Accepted Answer

Set an evaluation threshold alert on the online evaluation's groundedness metric so Foundry raises an alert (via Azure Monitor) when the aggregated score drops below the defined value

Answer

Increase the online evaluation sampling rate to 100% so every production request is scored and reviewed manually each day

Answer

Schedule a nightly batch evaluation job against a static golden test dataset and email the results to the team

Answer

Enable Application Insights distributed tracing on the agent endpoint and create a log query that counts tool-call spans