Question

Your RAG solution for a legal research assistant retrieves the top 20 chunks from an Azure AI Search hybrid index, but users report that highly relevant passages often appear far down the passed context and the LLM misses them. Retrieval recall is high, but the final answers lack precision. You must improve which chunks the LLM actually receives without re-indexing or changing chunk sizes. What should you add to the retrieval pipeline?

Accepted Answer

Insert a cross-encoder semantic reranker (reranking model) that rescores the 20 candidates and passes only the top 3–5 reordered chunks to the LLM.

Answer

Lower the similarity threshold so that more candidate chunks are returned before being sent to the LLM.

Answer

Switch the index from hybrid search back to pure keyword (BM25) search to boost exact term matches.

Answer

Increase the retrieval top-k from 20 to 50 so more relevant passages are guaranteed to be included in the prompt.