Question

A GenAI engineer has a customer-support RAG agent deployed with inference tables enabled. Product leadership wants a weekly quality report showing, for each request, whether the response was grounded in retrieved context and whether it correctly answered the question. The team has thousands of daily requests and only two subject-matter experts who can review a small sample manually. Which approach best scales the per-request quality assessment while keeping human effort focused where it matters most?

Accepted Answer

Run Mosaic AI Agent Evaluation with LLM-as-judge metrics (e.g., correctness and groundedness) across all logged requests, then route only low-scoring or low-confidence cases to the two SMEs for manual review.

Answer

Have the two SMEs manually label every request each week, since only human judgment can reliably assess groundedness and correctness at the required quality.

Answer

Rely solely on Lakehouse Monitoring drift metrics on the inference table to infer whether response quality is acceptable, avoiding any per-request scoring.

Answer

Compute average response latency and token count per request from the inference table and use those as proxies for answer quality in the weekly report.