Question

A GenAI engineer runs Mosaic AI Agent Evaluation on a customer-support RAG agent. The built-in judges (correctness, groundedness, relevance, safety) all pass, but the compliance team insists that every response must also avoid making promises about refund timelines, a domain-specific policy not captured by any built-in judge. The engineer wants this policy checked automatically on each evaluation run alongside the existing metrics. What is the most appropriate approach?

Accepted Answer

Define a custom LLM-judge metric with a guidelines/prompt describing the refund-timeline policy and pass it to mlflow.evaluate so it runs alongside the built-in judges.

Answer

Lower the safety judge's threshold so it also flags refund-timeline promises during evaluation.

Answer

Manually inspect every evaluation output for refund-timeline promises after each run and record findings in a spreadsheet.

Answer

Add the refund policy text to the system prompt and rely on the correctness judge to catch violations automatically.