Question

A Generative AI Engineer has run Mosaic AI Agent Evaluation on version 1 of a RAG chatbot and logged the results as an MLflow run. After changing the retriever's chunking strategy and re-running the evaluation on the same evaluation dataset, the engineer wants to determine whether the change improved or degraded answer quality before promoting the new version. What is the most effective way to make this decision using the evaluation results?

Accepted Answer

Compare the aggregate LLM-as-judge and retrieval metrics across the two MLflow evaluation runs to see whether scores like groundedness and retrieval relevance improved or regressed.

Answer

Deploy both versions to production behind a traffic split and rely on user thumbs-up/thumbs-down feedback to decide which version is better.

Answer

Inspect the inference tables from the production endpoint for the new version to measure latency and token cost differences.

Answer

Manually re-read a random sample of 5 generated answers from the new version without referencing the version 1 results.