Question

Your team runs automated evaluation on a customer-support agent in Azure AI Foundry. The built-in groundedness, relevance, and coherence evaluators are already configured, but the business requires a domain-specific check: each response must include a valid support ticket ID in a fixed format, and the evaluation must report the percentage of responses that pass this check across the test dataset. You need to add this measurement to the existing automated evaluation workflow with the least ongoing maintenance. What should you do?

Accepted Answer

Author a custom evaluator that implements the ticket-ID format check and returns a pass/fail score, then include it alongside the built-in evaluators in the evaluation run.

Answer

Switch the relevance evaluator to a higher threshold so that responses without a ticket ID are scored as failures.

Answer

Add a content safety risk evaluator to detect responses that omit the ticket ID.

Answer

Manually review the evaluation output CSV after each run and tally the responses that contain a valid ticket ID.