Question

Your team has a customer-support summarization prompt in production, backed by a stable test set of 200 representative cases with defined success metrics. Anthropic releases a newer model version, and a teammate proposes switching to it because it 'feels smarter' after trying a few chat examples. Before committing to the new model, what is the most responsible evaluation practice?

Accepted Answer

Run the existing 200-case eval set against the new model and compare metric scores to the current baseline before deciding

Answer

Deploy the new model immediately to all users since newer versions are always improvements

Answer

Build a brand-new test set specifically flattering to the new model's strengths, then measure it

Answer

Have the teammate try 10 more chat examples and switch if the outputs subjectively look better