Question

A Generative AI Engineer is building a RAG system over an internal knowledge base exported from a wiki. During inspection of the chunked output, they notice many chunks contain only navigation breadcrumbs, page footers with copyright notices, empty table cells, or single-word headers with no surrounding content. These chunks are being embedded and indexed alongside meaningful content, and retrieval quality has degraded because irrelevant chunks are frequently returned. What is the most effective step to take before generating embeddings and populating the Vector Search index?

Accepted Answer

Apply content-quality filters that drop chunks below a minimum token/character threshold and remove boilerplate patterns (breadcrumbs, footers, empty cells) prior to embedding

Answer

Increase the chunk size so that low-value fragments are merged into larger chunks that also contain meaningful text

Answer

Switch to a higher-dimensional embedding model so the vector representation better distinguishes low-value chunks at query time

Answer

Add a reranker after retrieval to push the boilerplate chunks lower in the result ordering