Question

A team uses an LLM-as-a-judge approach to score the quality of summaries produced by their production summarization service. They notice that the judge model consistently assigns higher scores to longer summaries even when shorter ones are more accurate, and that re-running the same evaluation produces noticeably different scores each time. The team needs to make the automated evaluation more reliable and less biased before using it as a release gate. Which combination of changes BEST addresses both problems?

Accepted Answer

Set the judge model's temperature to 0 (or near 0) for deterministic scoring, and add explicit rubric criteria in the judge prompt that separate factual accuracy from length/verbosity to counteract length bias.

Answer

Increase the judge model's temperature and average scores across many samples, then instruct the judge to always prefer the more detailed answer.

Answer

Replace the LLM judge with BLEU and ROUGE n-gram overlap metrics only, since these are fully deterministic and require no reference-free reasoning.

Answer

Switch to a larger judge model and provide the two candidate summaries in a fixed order every time so the model's ordering preference stabilizes the scores.