Question

A team is building a customer-facing assistant that drafts support replies. Leadership says the replies must be 'good.' Before writing any evals, the team wants to translate this into something they can actually measure. Which approach best defines useful success criteria for their evaluation?

Accepted Answer

Specify concrete, measurable targets across the dimensions that matter—such as factual accuracy against the knowledge base ≥95%, correct tone on a rubric, and no policy violations—each with a threshold.

Answer

Ask several team members to read a handful of outputs and give a thumbs-up or thumbs-down based on overall impression.

Answer

Set a single target of 'user satisfaction should improve' and check it after the assistant has been live for a quarter.

Answer

Adopt whatever benchmark scores the model reports on public leaderboards as the definition of quality for this use case.