Question

A team maintains a customer-support summarization prompt in production. Before shipping any prompt change, they want a reliable way to confirm the new version does not degrade quality compared to the current version. They have 200 representative support transcripts. Which approach best supports catching quality regressions when they update the prompt?

Accepted Answer

Establish scored baseline results for the current prompt on all 200 transcripts using a fixed grading rubric, then re-run and compare the new prompt's scores against that baseline before shipping

Answer

Run the new prompt on the 200 transcripts, spot-check a handful of outputs by reading them, and ship if those few look reasonable

Answer

Increase the temperature on the new prompt so it produces more varied summaries, making regressions easier to notice by eye

Answer

Ask the model to grade its own new summaries and approve the change if the model reports high confidence