Question

A Generative AI Engineer has built a RAG application over an internal knowledge base and wants to quantitatively evaluate whether the vector search retriever is returning relevant documents before investing in prompt tuning. The team can manually label a small set of representative questions with the specific source chunks that should be retrieved for each. Which evaluation approach best measures retrieval relevance in this situation?

Accepted Answer

Build a golden evaluation dataset of question-to-relevant-chunk pairs and compute retrieval metrics such as recall@k and precision@k against the retriever's returned results

Answer

Measure the average cosine similarity score between each query embedding and its top returned chunk, treating higher scores as proof of correct retrieval

Answer

Send the final generated answers to an LLM judge that scores fluency and grammar, using those scores as a proxy for retrieval quality

Answer

Track the end-to-end latency of the vector search endpoint and use lower latency as the indicator that retrieval is returning the most relevant chunks