Question

A product team has deployed a gen AI assistant that drafts customer email responses. Leadership wants a reliable way to measure whether output quality is actually improving as the team iterates on prompts and grounding. Automated metrics alone haven't correlated well with what customers consider a 'good' reply. What is the most effective approach to evaluate and improve output quality in this situation?

Accepted Answer

Establish a structured evaluation rubric scored by human reviewers against defined criteria (accuracy, tone, completeness) and track scores across iterations

Answer

Increase the model's temperature setting so responses become more varied and natural-sounding

Answer

Rely solely on the number of tokens generated as a proxy for how thorough each reply is

Answer

Switch to the largest available foundation model, since bigger models always produce higher-quality output