Question

A legal-tech company is building an assistant that answers questions across a library of 50,000 contract documents that grows weekly. An engineer proposes pasting the full text of all potentially relevant documents directly into the prompt each time using a very large context window model. A team lead objects that this approach will not scale. Which single design choice best addresses the scalability concern while keeping answers grounded in the correct documents?

Accepted Answer

Implement a Retrieval Augmented Generation (RAG) pipeline that retrieves only the most relevant document chunks from a vector store and injects them into the prompt

Answer

Fine-tune the foundation model on all 50,000 documents so it memorizes the contract content and no context needs to be passed at query time

Answer

Increase the model's max token output limit so it can process more documents per request

Answer

Lower the temperature parameter to 0 so the model reads all documents more efficiently