Question

A company deployed a customer-facing chatbot built on a foundation model. Automated metrics show the responses are grammatically correct and factually accurate, but the product team is concerned about whether responses feel empathetic, on-brand, and genuinely helpful to real customers. Which evaluation approach is MOST appropriate to assess these qualities before a wider rollout?

Accepted Answer

Human evaluation, where reviewers rate sample responses on subjective dimensions like tone, empathy, and helpfulness

Answer

Increase the model's temperature parameter and re-run the automated accuracy tests

Answer

Calculate BLEU scores between the chatbot output and a reference dataset of correct answers

Answer

Measure average inference latency and token throughput under load