Question

A financial services company is building a customer support summarization feature on Amazon Bedrock. The compliance team requires that evaluation reflect their own domain-specific rubric (correct handling of regulatory disclaimers, factual accuracy of account figures, and tone) rather than generic quality scores. They have a labeled dataset of 500 real support transcripts with reference summaries and want repeatable, automated scoring against these criteria before each model version is promoted. Which approach best meets these requirements using Bedrock model evaluation?

Accepted Answer

Create a Bedrock model evaluation job using an LLM-as-a-judge (model-based) evaluator with a custom metric prompt encoding the rubric, supplying the company's own labeled dataset as the evaluation input.

Answer

Run a Bedrock automatic model evaluation job using only the built-in accuracy and robustness metrics with the default sample prompt datasets.

Answer

Set up a Bedrock human evaluation job and have compliance analysts manually score every model version, treating this as the only repeatable mechanism.

Answer

Enable Bedrock model invocation logging to CloudWatch and compute BLEU scores manually against the reference summaries in a spreadsheet.