Question

Your team runs automated evaluations on Claude's customer-email drafts using another Claude call as an LLM-based grader. Reviewers notice the grader assigns wildly different scores to nearly identical drafts across runs, making it hard to trust regression results. What change would most improve the reliability of these automated grades?

Accepted Answer

Give the grader a detailed scoring rubric with explicit criteria and examples, and set a low temperature for the grading call

Answer

Increase the temperature of the grading call so the grader considers more diverse interpretations of quality

Answer

Reduce the test set to only the three drafts reviewers agree are perfect examples

Answer

Replace the LLM grader with a single human reviewer who scores drafts whenever they have time