Question

A GenAI engineer configures an LLM-as-judge metric in Mosaic AI Agent Evaluation to score answer correctness for a customer support RAG assistant. After the first evaluation run, stakeholders complain that the judge's scores don't match their intuition — several responses they consider clearly wrong were rated as correct. Before trusting the judge for ongoing quality gating, what is the most appropriate next step?

Accepted Answer

Assemble a small set of human-labeled examples and measure the LLM judge's agreement with those labels, then refine the judge's grading prompt/criteria until alignment is acceptable

Answer

Increase the judge model's temperature so it produces more varied scores that better reflect the range of answer quality

Answer

Switch the correctness metric to a purely lexical overlap score like BLEU, since LLM judges are inherently unreliable for correctness

Answer

Run the evaluation multiple times and average the scores to eliminate the disagreement with stakeholder intuition