Question

A company is deploying a customer-facing chatbot built on a foundation model. They have a set of 500 reference question-answer pairs and want to measure how closely the model's responses match the expected answers at scale, cheaply, and repeatably during each release cycle. However, they are concerned that automated similarity scores alone may miss subtle problems with tone and empathy. Which evaluation approach best balances these needs?

Accepted Answer

Use automated metrics against the reference dataset for broad, repeatable regression checks, and supplement with periodic human evaluation on a sampled subset to assess tone and empathy.

Answer

Use only automated metrics against the reference answers for every release, and skip human review entirely to save cost.

Answer

Rely exclusively on human reviewers to score all 500 responses on every release to guarantee the highest quality.

Answer

Measure only inference latency and token cost per response, since those are the primary indicators of chatbot quality.