Question

Your team is iterating on a prompt for a customer-support summarization feature. A teammate proposes deciding whether each new prompt version is better by having two engineers read a handful of outputs and share their gut reaction in a Slack thread. As the person responsible for quality, what is the strongest objection to this approach and the better alternative?

Accepted Answer

Ad hoc 'vibe checks' on a few outputs are not repeatable or comprehensive; instead build a representative test set with defined success criteria and score each prompt version against it.

Answer

The approach is fine as long as the two engineers are senior enough; seniority is what makes manual review reliable at scale.

Answer

Manual review is too slow, so the team should skip evaluation entirely and ship whichever version was written most recently.

Answer

The real fix is to raise the temperature so outputs become more varied, making quality differences easier for reviewers to notice.