The model doesn't know your documents
A language model knows what was in its training data, not your contracts, manuals, or support tickets. You can't paste everything into every prompt either: it's too much, too slow, and too expensive. Retrieval-augmented generation, RAG, solves that by finding the few passages that matter for each question and giving only those to the model.
Done well, it's the backbone of most useful internal assistants. Done badly, it's why they confidently answer from the wrong page.
Filing by meaning
First, documents are cut into chunks, a paragraph or a section each. Each chunk is turned into an embedding: a list of numbers where passages about similar things end up pointing in similar directions. Real embeddings have hundreds of dimensions; the figure uses a toy with two, so direction on the round index is meaning.
When a question comes in, it gets embedded the same way, and retrieval looks for the chunks pointing most nearly in the same direction. That closeness is measured with cosine similarity. In the figure, the top three score 0.994, 0.951 and 0.868, from two of the three documents. The next one, at 0.788, doesn't make the cut.
Answer from the sources, and show them
The retrieved chunks are clipped to the question and handed to the model with an instruction to answer only from them, and to say which ones it used. The citations are what make the system trustworthy: a person can click through and check, and you can measure whether answers come from the right places.
Where RAG systems usually go wrong
- Chunks that are too big bury the answer; too small and they lose context. Test sizes on real questions.
- Pure vector search misses exact terms like part numbers and names. Combine it with keyword search.
- No permissions: retrieval must only return chunks the person asking is allowed to see.
- No evaluation: keep a set of real questions with known source passages and measure retrieval on it.
- Stale indexes: documents change, so re-embed on update, not once at launch.