Question

A team building a customer-support summarization feature achieves 95% on their automated eval, which grades summaries by ROUGE overlap against reference summaries. However, support agents complain that the deployed summaries frequently omit the customer's actual requested action, making them useless in practice. What is the most likely root cause and the best corrective step?

Accepted Answer

The eval metric measures surface word overlap rather than whether the required outcome (capturing the requested action) is present; redefine the success criteria and add a targeted check that verifies the requested action is captured.

Answer

The model temperature is too high, causing inconsistent summaries; lower it to 0 and rerun the same eval.

Answer

The test set is too small; add more examples and the ROUGE score will better reflect real usefulness.

Answer

The prompt lacks a role assignment; give Claude a 'senior support analyst' persona and the omission problem will resolve.