Question

Your team is preparing to evaluate a customer-support RAG agent in Azure AI Foundry before production release. You have only a small set of real user questions and need a larger, diverse test dataset that includes adversarial and edge-case queries so that safety and quality evaluators produce statistically meaningful results. You want to minimize the manual effort of hand-writing hundreds of prompts. Which approach best generates the required test dataset?

Accepted Answer

Use the Azure AI Evaluation SDK simulator (adversarial and non-adversarial simulators) to synthesize a diverse set of query/response pairs, then run the evaluators against the generated dataset.

Answer

Enable production traffic sampling first, then wait to collect enough real conversations before running any evaluation.

Answer

Duplicate the small set of real questions multiple times until you reach several hundred rows and run the evaluators on the expanded file.

Answer

Deploy the agent to a provisioned throughput (PTU) endpoint so the higher throughput automatically increases the size of the evaluation dataset.