Question

Your team has deployed a customer-support RAG agent to production in Azure AI Foundry. Leadership wants ongoing assurance that live responses remain grounded and relevant without manually assembling a test dataset each time. You need an approach that automatically evaluates a sample of real production traffic on a recurring basis and surfaces the AI quality metrics in the monitoring dashboard. Which approach should you configure?

Accepted Answer

Enable continuous evaluation (online evaluation) in Foundry observability, sampling a percentage of production traces and running groundedness and relevance evaluators on a schedule.

Answer

Create a one-time batch evaluation run against a static golden test dataset and rerun it manually before each release.

Answer

Configure a data drift monitor on the embedding inputs to detect when response quality degrades.

Answer

Add a content safety filter to the serverless endpoint to block harmful outputs and report quality scores.