Question

A Generative AI Engineer has built a RAG pipeline over a corpus of internal knowledge base articles. During evaluation, users report that retrieved chunks are often topically related to the query but frequently lack the specific detail needed to answer the question, forcing the LLM to say it cannot find the answer. Inspection shows each chunk is 1500 tokens with no overlap, and many chunks combine an FAQ heading with several unrelated sub-answers. What is the most effective first change to improve retrieval relevance?

Accepted Answer

Reduce chunk size and add semantic-aware splitting so each chunk captures a single coherent topic, with a small token overlap between adjacent chunks

Answer

Switch to a higher-dimensional embedding model while keeping the same 1500-token chunks

Answer

Increase the number of retrieved chunks (top-k) from 3 to 20 so more context reaches the LLM

Answer

Re-index the vector store more frequently using Delta Sync to ensure freshness of the embeddings