Question

A developer is preparing a Bedrock model evaluation job for a customer-facing chatbot built on a foundation model. The team must measure two things before launch: (1) how factually accurate and helpful the responses are according to subject-matter experts who understand the company's specialized insurance domain, and (2) an objective, repeatable toxicity score that can be re-run automatically on every model version as part of the release pipeline. The team wants to minimize ongoing operational effort for the repeatable check while still capturing nuanced domain-quality judgments. Which approach best satisfies both requirements?

Accepted Answer

Use a human evaluation job (with your own work team) for the domain accuracy/helpfulness assessment, and an automatic model evaluation job for the toxicity metric that runs on each release.

Answer

Use an automatic model evaluation job with built-in metrics (including toxicity) for both requirements, since automatic metrics are cheaper and can be scheduled.

Answer

Use a human evaluation job for both requirements because only human reviewers can reliably judge toxicity and factual accuracy in a specialized domain.

Answer

Use an automatic evaluation job for domain accuracy and a human evaluation job for toxicity, so experts can flag offensive content the automated metrics miss.