Question

A product team at an insurance company has deployed a gen AI assistant that answers policy questions. They frequently adjust prompts to improve responses, but they worry that a change fixing one type of question might silently break answers to other questions. What practice should they adopt to reliably measure whether each prompt change improves overall output quality?

Accepted Answer

Maintain a curated evaluation dataset of representative questions with expected answers, and re-run it against the model after every prompt change to compare results

Answer

Increase the model's temperature setting so responses become more varied and cover more cases

Answer

Rely on individual team members spot-checking a few answers each time they change a prompt

Answer

Fine-tune the model on the full policy documents so prompt changes are no longer necessary