Question

A media company deploys a gen AI tool that summarizes news articles. Early users complain that some summaries are fluent and well-written but occasionally contradict details in the original article. The product team wants to measure this specific problem systematically before their next model update. Which evaluation approach most directly targets the issue they are seeing?

Accepted Answer

Score each summary for factual consistency (faithfulness) against its source article, using human raters or an automated grounding check

Answer

Measure the average response latency of the summarization tool across a large sample of articles

Answer

Count the number of tokens in each summary to confirm outputs stay within a fixed length budget

Answer

Survey users on how readable and grammatically polished they find the summaries