Question

A team has been improving a prompt for classifying customer support tickets into 12 categories. So far they have judged quality by manually reading 5 outputs after each prompt change and deciding whether it 'feels better.' They now want a repeatable way to know whether a change genuinely improves accuracy without introducing regressions in less common categories. Which change to their process best addresses this?

Accepted Answer

Build a labeled test set of representative tickets covering all 12 categories (including rare ones) with known-correct answers, and score each prompt version against it automatically.

Answer

Increase the temperature so the model explores more classification options, then keep reviewing 5 outputs by hand each time.

Answer

Keep the manual spot-check but add a second reviewer so two people read the same 5 outputs and average their impressions.

Answer

Switch to a larger model, which will make classifications accurate enough that a formal test set becomes unnecessary.