Question

A data science team is building a document summarization feature on Amazon Bedrock. They have a curated dataset of 500 source documents, each paired with a human-written reference summary. They want to run an automated Bedrock model evaluation job to compare two candidate foundation models and objectively measure how closely each model's generated summaries match the reference summaries, without requiring human reviewers for this initial screening. Which evaluation approach and metric should they configure?

Accepted Answer

Configure an automatic model evaluation job using the Text summarization task type with a BERTScore metric and provide the prompt dataset that includes the reference (ground truth) summaries.

Answer

Configure a human evaluation job using the Text summarization task type, since summary quality cannot be measured without human raters, and provide the reference summaries as scoring rubrics.

Answer

Configure an automatic model evaluation job using the General text generation task type with a Toxicity metric, since summaries must be safe before quality is assessed.

Answer

Configure a Bedrock Guardrails contextual grounding policy on both models and compare the grounding scores across the 500 documents as the summarization quality metric.