Question

Your team built an evaluation set of 200 examples to measure the quality of a document-summarization prompt. Each summary can be correct in many valid phrasings, so exact string matching against a reference summary produces misleadingly low scores. You need a grading method that scores summaries reliably at this scale without a human reviewing all 200 outputs each run. Which grading approach best fits this evaluation?

Accepted Answer

Use an LLM-based grader ('model-graded eval') with a clear rubric that checks whether each summary captures the required key points and stays faithful to the source

Answer

Keep exact-match scoring but lower the pass threshold so more summaries count as correct

Answer

Drop the evaluation set and rely on the team informally reading a few outputs to judge whether the prompt 'feels' good

Answer

Have one engineer manually grade all 200 summaries by hand on every prompt iteration