Question

Your team built an eval suite for a customer-support assistant that generates free-form answers to billing questions. You chose exact-string match against golden answers as your grading metric. In practice, the assistant produces correct, well-phrased responses that differ in wording from your golden answers, so your eval reports a very low pass rate even though humans judge the outputs as good. What is the best fix?

Accepted Answer

Switch the grading approach to a semantic or LLM-based rubric grader that scores whether the response conveys the correct information, rather than requiring identical wording

Answer

Delete the golden answers and rely on periodic manual spot-checks by the team instead of an automated eval

Answer

Lower the temperature to 0 so the model always produces the exact wording in your golden answers

Answer

Reduce max_tokens so responses are shorter and more likely to match the golden answers exactly