Question

A financial services company runs a RAG-based customer support assistant on Amazon Bedrock. Legal has flagged that the assistant occasionally produces answers containing facts that are not supported by the retrieved source documents, even when retrieval returns relevant passages. The team wants an automated, offline evaluation metric they can compute over a labeled test set to specifically quantify how often the generated answer contains claims that are NOT grounded in the retrieved context. Which evaluation approach most directly measures this problem?

Accepted Answer

Compute a faithfulness (groundedness) metric that decomposes the generated answer into claims and verifies each claim against the retrieved context.

Answer

Compute a context-recall metric that measures whether all relevant ground-truth passages were retrieved from the vector store.

Answer

Compute a ROUGE-L score comparing the generated answer against the reference answer to measure lexical overlap.

Answer

Compute an answer-relevance metric that scores how well the generated answer addresses the user's original question.