Question

A team maintains a customer-support summarization prompt that scores 92% on their curated evaluation set of 200 labeled examples. A developer proposes a prompt change that they believe reads more clearly and 'feels better' in a few manual spot-checks. Before deploying the change to production, what is the most responsible next step consistent with an evals-driven workflow?

Accepted Answer

Re-run the full evaluation set against the new prompt and compare scores to the 92% baseline before deciding to deploy

Answer

Deploy the new prompt immediately since it reads more clearly and the developer's spot-checks looked good

Answer

Add the new prompt's spot-check outputs to the evaluation set as additional ground-truth examples

Answer

Increase the temperature on the new prompt so its outputs vary more and cover more cases during spot-checks