Question

A Generative AI Engineer is building a RAG application over a corpus of internal support articles that were exported from three different systems. During data profiling, they discover that roughly 30% of the articles are near-duplicates (same content with minor formatting differences) because articles were migrated multiple times. Early retrieval tests show that the top-k results are frequently filled with several copies of the same article, crowding out other relevant content. What is the most effective step to improve retrieval quality before generating embeddings?

Accepted Answer

Deduplicate the source documents during preprocessing (e.g., using content hashing or similarity-based clustering) so only one canonical version of each article is embedded and indexed

Answer

Increase the number of retrieved documents (k) so that non-duplicate content is more likely to appear in the results

Answer

Switch to a higher-dimensional embedding model so duplicate articles produce more distinct vectors

Answer

Reduce the chunk size so that duplicate articles are split into smaller, less-overlapping segments