Question

A team runs Mosaic AI Agent Evaluation nightly on a golden dataset of 500 questions for their production RAG assistant. After a recent change that swapped the generation model to a larger foundation model, the evaluation report shows the average correctness and groundedness scores each improved by about 3%, but the total_input_token_count and total_output_token_count metrics roughly tripled, and mean response latency rose from 1.2s to 3.8s. The product owner requires sub-2s p90 latency and a fixed monthly serving budget. What is the most appropriate way to use these evaluation results to decide next steps?

Accepted Answer

Weigh the marginal 3% quality gain against the tripled token cost and latency SLA violation, and either revert or test cheaper alternatives (smaller model, tighter prompts, fewer retrieved chunks) that meet the latency and budget constraints.

Answer

Keep the larger model because any improvement in correctness and groundedness always outweighs cost and latency regressions in a production assistant.

Answer

Disable the cost and latency metrics in Agent Evaluation so the report focuses only on correctness and groundedness, which are the metrics that matter for quality.

Answer

Ignore the offline evaluation entirely and rely solely on production inference table monitoring, since offline token counts are not representative of real traffic.