Question

A financial services company is building a customer support summarization feature on Amazon Bedrock. Before committing to a single foundation model, the ML team wants to systematically compare three candidate models on the same set of 500 representative support-ticket summaries, scoring each model's output for relevance, coherence, and factual accuracy against reference answers. They want a repeatable, managed process that produces quantitative metrics rather than ad-hoc manual testing. Which approach best meets these requirements with the least custom engineering?

Accepted Answer

Use Amazon Bedrock Model Evaluation jobs, providing a custom prompt dataset with reference responses and selecting automatic and human-based evaluation metrics to score each candidate model.

Answer

Deploy each model behind a separate SageMaker endpoint and write a Lambda function that computes BLEU scores against the reference answers.

Answer

Enable Amazon Bedrock model invocation logging to CloudWatch and manually inspect the logged outputs for each model in the console.

Answer

Fine-tune all three models on the 500 tickets, then compare their post-training validation loss to pick the best summarizer.