Question

A financial services company is building a RAG-based internal assistant that answers employee questions about corporate HR policies. Compliance requires that the assistant NEVER fabricate policy details and must clearly refuse when a question falls outside the indexed policy documents. During testing, the LLM sometimes confidently answers questions about topics not present in the retrieved context. Which design change most directly addresses this requirement?

Accepted Answer

Add a grounding instruction to the system prompt directing the model to answer only from the provided context and to respond with a refusal when the context is insufficient

Answer

Increase the number of retrieved chunks (top-k) so more context is always passed to the model

Answer

Switch to a larger foundation model with more parameters to improve factual accuracy

Answer

Lower the embedding model's dimensionality to speed up retrieval and reduce noise