Question

Your team runs an Azure AI Foundry customer-support agent that must always include a legal disclaimer phrase in responses that mention refund policies. The built-in AI-assisted quality metrics (groundedness, relevance, coherence, fluency) do not capture this business rule. You need the automated batch evaluation workflow to flag every response missing the required phrase, in addition to the built-in metrics. What should you do?

Accepted Answer

Author a custom code-based evaluator that programmatically checks each output for the required phrase and register it alongside the built-in evaluators in the evaluation run.

Answer

Increase the groundedness metric threshold so responses missing the disclaimer score lower and are automatically flagged.

Answer

Enable the risk and safety evaluations for harmful content, which will detect the missing disclaimer as a compliance violation.

Answer

Add the disclaimer requirement as a system prompt instruction and rely on the coherence metric to confirm it was followed.