Question

A Generative AI Engineer has built a RAG-based customer support agent and wants to systematically compare two candidate prompt templates before promoting either to production. They have assembled a curated dataset of representative questions, each with a reference (ground-truth) answer and the relevant supporting documents. They want to run Mosaic AI Agent Evaluation to measure answer correctness, groundedness, and retrieval relevance for both templates, then choose the better one. Which approach correctly uses Agent Evaluation for this offline comparison?

Accepted Answer

Run mlflow.evaluate() with the Agent Evaluation harness on the curated evaluation dataset for each template, comparing the built-in LLM-judge metrics (correctness, groundedness, retrieval relevance) across runs.

Answer

Deploy both templates to production serving endpoints first, then compare their inference table logs after a week of real traffic to decide which is better.

Answer

Enable Lakehouse Monitoring on the source Delta tables and use its data drift metrics to determine which template produces higher-quality answers.

Answer

Skip the labeled dataset and rely solely on measuring p95 latency and cost-per-request for each template, selecting whichever is cheaper and faster.