Question

Your team maintains a customer-facing summarization assistant with an automated eval suite of 200 test cases that all pass with high scores. After deployment, users report the assistant occasionally invents product features that don't exist when summarizing sparse or off-topic support tickets. Your current test set contains only well-formed, feature-rich tickets. What is the most effective next step to improve your evaluation process?

Accepted Answer

Add representative test cases covering sparse, ambiguous, and off-topic tickets—including expected 'no relevant features found' outputs—so the eval catches the failure mode

Answer

Raise the passing score threshold on the existing 200 test cases so the eval is stricter overall

Answer

Switch from automated grading to manually reviewing every production output before it reaches users

Answer

Increase the model's temperature so summaries have more variety and feel less repetitive