A language model on its own answers from training data, confidently and sometimes wrongly. Retrieval-augmented generation retrieves relevant passages from your own approved content first, then asks the model to answer using only those passages, with citations.
The pipeline
- Split documents into passages that stand alone
- Generate embeddings and store them in a vector index
- Retrieve the most relevant passages for each question
- Provide them as context with instructions to cite
- Return the answer with links to sources
Retrieval quality decides everything
Most disappointing RAG systems have a retrieval problem, not a model problem. If the right passage is not retrieved, no model can answer correctly. Invest in chunking strategy, metadata filters and hybrid keyword-plus-vector search before changing models.
Permissions must be respected
Filter by the asking user's access rights at retrieval time. A system that surfaces documents someone should not see is a data breach with a friendly interface.
Keep the index fresh
Re-index on document change rather than on a schedule, and show the source date in the answer so people can judge currency themselves.
Measure it honestly
Build an evaluation set of real questions with known answers. Track retrieval accuracy separately from answer quality so you know which half to fix.
If the answer cannot be traced to a source document, it should not be presented as an answer.
Where to start
Collect fifty real questions your team gets asked, and check whether your retrieval step surfaces the right document for each. That number is your ceiling.
