Question

You maintain a production RAG assistant for an insurance company. After tuning chunk sizes and embedding models, you propose switching from pure semantic retrieval to hybrid (semantic + keyword) retrieval, expecting better answers on policy-number lookups without regressing general questions. Leadership wants statistically sound evidence before the change is promoted. Which approach best validates the improvement before full rollout?

Accepted Answer

Route a portion of live traffic across two deployments (semantic vs. hybrid) in an A/B test, capture retrieval relevance and groundedness metrics per variant, and compare results before promoting the winner.

Answer

Run an offline batch evaluation of the hybrid pipeline on a labeled golden dataset, then deploy the hybrid retriever to 100% of traffic immediately since offline scores improved.

Answer

Increase the similarity threshold on the existing semantic retriever until policy-number queries return fewer irrelevant chunks, avoiding any change to the retrieval strategy.

Answer

Enable content safety filtering on the endpoint and monitor blocked-response rates as a proxy for retrieval quality between the two strategies.