Testing, Validation, and Troubleshooting

A developer is preparing a Bedrock model evaluation job for a customer-facing chatbot built on a foundation model. The team must measure two things before launch: (1) how factually accurate and helpful the responses are according to subject-matter experts who understand the company's specialized insurance domain, and (2) an objective, repeatable toxicity score that can be re-run automatically on every model version as part of the release pipeline. The team wants to minimize ongoing operational effort for the repeatable check while still capturing nuanced domain-quality judgments. Which approach best satisfies both requirements?

A financial services company is building a customer support summarization feature on Amazon Bedrock. The compliance team requires that evaluation reflect their own domain-specific rubric (correct handling of regulatory disclaimers, factual accuracy of account figures, and tone) rather than generic quality scores. They have a labeled dataset of 500 real support transcripts with reference summaries and want repeatable, automated scoring against these criteria before each model version is promoted. Which approach best meets these requirements using Bedrock model evaluation?

A data science team is building a document summarization feature on Amazon Bedrock. They have a curated dataset of 500 source documents, each paired with a human-written reference summary. They want to run an automated Bedrock model evaluation job to compare two candidate foundation models and objectively measure how closely each model's generated summaries match the reference summaries, without requiring human reviewers for this initial screening. Which evaluation approach and metric should they configure?

A GenAI customer-support application invokes an Amazon Bedrock model through a Lambda function. Traffic follows a strong daily pattern: high volume during business hours and near-zero overnight. The team wants to be alerted when the application's error rate (failed invocations) deviates significantly from its normal behavior, but a static CloudWatch alarm threshold keeps producing false alarms overnight (when a single error looks like a 100% error rate) and misses moderate spikes during peak load. What is the MOST appropriate way to configure the alerting?

A team runs a production customer-support chatbot on Amazon Bedrock using a specific foundation model version. Bedrock announces a newer model version, and the team wants to adopt it. Leadership requires that the new model be validated against real production traffic patterns without exposing customers to any degraded responses before a full cutover. Which approach best satisfies this requirement while allowing objective comparison of output quality?

A financial analytics team runs a customer-facing chatbot on Amazon Bedrock that uses tool-use (function calling) to fetch account balances. In production, roughly 3% of model turns cause the downstream Lambda parser to throw an exception because the model occasionally returns a tool-use block with slightly malformed or extra prose around the JSON arguments. These failures currently return a raw stack trace to the user and are hard to diagnose because there is no correlation between the failing request and the model output that caused it. The team wants to make the application resilient and debuggable without retraining or changing the model. Which approach BEST addresses both the resilience and the diagnosability requirements?

A financial services company deploys a Bedrock-based document classification application that returns a category label along with a confidence score (0.0–1.0) for each prediction. Downstream automated workflows only auto-approve documents when the model reports confidence above 0.9; everything else is routed to human reviewers. During validation, the QA team notices that many predictions scored above 0.9 are actually wrong, causing incorrect auto-approvals. They want a metric that specifically quantifies how well the model's reported confidence scores match the actual likelihood of correctness, so they can decide whether to trust the threshold. Which evaluation approach BEST addresses this need?

A team runs an automated Bedrock model evaluation job to compare two foundation models for a news-article summarization feature. Their evaluation dataset uses reference summaries written by editors, but the two models express the same key facts using different wording and sentence structures. The team notices that a model producing summaries they consider high-quality receives surprisingly low scores on the current metric, because it rarely reuses the exact tokens or n-grams from the reference text. Which change to their evaluation approach best addresses this problem while remaining automated?

A team runs a customer-facing FAQ assistant built on Amazon Bedrock. During QA, testers notice that when users phrase the same underlying question in slightly different ways (e.g., 'How do I reset my password?' vs 'I forgot my login credentials, what now?'), the assistant sometimes returns correct answers and other times returns 'I don't have that information.' Accuracy on their fixed golden-dataset regression suite remains high (94%), so the issue is not caught before deployment. Which testing enhancement most directly addresses this observed inconsistency?

A team uses an LLM-as-a-judge approach to score the quality of summaries produced by their production summarization service. They notice that the judge model consistently assigns higher scores to longer summaries even when shorter ones are more accurate, and that re-running the same evaluation produces noticeably different scores each time. The team needs to make the automated evaluation more reliable and less biased before using it as a release gate. Which combination of changes BEST addresses both problems?

A GenAI developer operates a production RAG chatbot built on API Gateway, Lambda, a Bedrock Knowledge Base, and Amazon Bedrock foundation models. Users report intermittent high end-to-end latency, but the team cannot determine whether the delay originates from vector retrieval, model invocation, or the Lambda orchestration code. They need a solution that provides a per-request breakdown of time spent in each downstream call with minimal custom instrumentation. Which approach best addresses this need?

A GenAI team runs a customer support chatbot on Amazon Bedrock. Finance reports that daily inference costs occasionally spike 5x with no corresponding increase in user traffic. The team suspects certain conversations are consuming abnormally large token counts, but current monitoring only shows aggregate invocation counts and total cost. They need to identify which specific requests drive the cost spikes and correlate them with prompt content so they can add controls. What is the MOST effective approach to gain this visibility?

A team runs a customer-support chatbot built with API Gateway, Lambda, and Amazon Bedrock. Occasionally a user reports a garbled or incomplete answer, but engineers cannot reconstruct which model invocation, retrieved documents, and prompt version produced that specific response. They want an observability approach that lets them trace a single end-user request across all components and inspect the exact Bedrock input/output for that request. Which approach best meets this requirement with the least custom instrumentation?

A team runs an Amazon Bedrock human evaluation job to assess the tone and helpfulness of customer-support responses from two candidate models. Each response is rated on a 5-point Likert scale by multiple work-team members. After the job completes, the team notices that individual reviewers frequently assign very different scores to the same response, making the aggregate rankings unreliable. Which action most directly addresses the root cause of this problem?

A team runs an evaluation on their RAG-based customer support assistant. Using an evaluation framework, they observe that the 'context recall' metric is high (retrieved chunks reliably contain the information needed to answer), but the 'answer relevance' metric is low (generated responses often drift off-topic or fail to directly address the user's question). The retrieval pipeline appears healthy. Which action most directly targets the observed problem?

A financial services company runs a RAG-based customer support assistant on Amazon Bedrock. Legal has flagged that the assistant occasionally produces answers containing facts that are not supported by the retrieved source documents, even when retrieval returns relevant passages. The team wants an automated, offline evaluation metric they can compute over a labeled test set to specifically quantify how often the generated answer contains claims that are NOT grounded in the retrieved context. Which evaluation approach most directly measures this problem?

A team maintains a RAG-based customer support assistant on Amazon Bedrock. Each week they update the knowledge base with new product documentation and occasionally switch the retrieval reranking configuration. After a recent update, users reported that previously correct answers had degraded, but the change was only noticed days later through complaints. The team wants to automatically detect answer-quality regressions before each release reaches production. Which approach BEST addresses this requirement?