Question

A team maintains a RAG-based customer support assistant on Amazon Bedrock. Each week they update the knowledge base with new product documentation and occasionally switch the retrieval reranking configuration. After a recent update, users reported that previously correct answers had degraded, but the change was only noticed days later through complaints. The team wants to automatically detect answer-quality regressions before each release reaches production. Which approach BEST addresses this requirement?

Accepted Answer

Maintain a curated golden dataset of representative questions with expected reference answers, and run an automated evaluation (using an LLM-as-a-judge and similarity scoring) against each release candidate in the CI/CD pipeline, blocking promotion if scores fall below a baseline threshold.

Answer

Enable Bedrock model invocation logging to CloudWatch and set an alarm on the total number of InvokeModel calls per day so spikes indicate quality problems.

Answer

Rely on a Bedrock Guardrail with contextual grounding enabled to reject any low-quality answer at inference time, eliminating the need for pre-release testing.

Answer

Schedule periodic human review of a random sample of production conversations each month and file tickets when reviewers find incorrect answers.