RAG and grounding: giving language models the right documents
Retrieval-augmented generation puts relevant documents into the prompt so the model answers from them. This page covers each part of a RAG system and the variants that fix its weak spots.
Overview
A language model's own knowledge is frozen at training time and can be wrong. RAG fetches passages from your documents at question time and asks the model to answer from them, with citations. It is the most common way to build assistants over private or changing information.
The pipeline
| Step | Choice to make | Read more | Try it |
|---|---|---|---|
| Chunking | Size and boundaries of passages | Simulation | |
| Embedding | Model that turns text into vectors | Embeddings | Simulation |
| Index | Vector database and index type | HNSW, pgvector, Qdrant | Simulation |
| Retrieval | Vector, keyword or both | Hybrid search, BM25 | |
| Re-ranking | Reorder the top results precisely | Cross-encoder reranking | |
| Generation | Prompt with passages and citation rules | Prompt engineering | |
| Evaluation | Faithfulness, relevance, recall | RAGAS | Simulation |
Variants that fix common failures
| Failure | Fix | Read more |
|---|---|---|
| The question does not match document wording | Rewrite or expand the query, or embed a hypothetical answer | Query rewriting, HyDE |
| Chunks lose their context | Prepend document context to each chunk | Contextual retrieval |
| Wrong passages retrieved | Check relevance and retry or search the web | Advanced RAG |
| Questions need several hops | Let an agent plan searches | Agentic RAG |
| Answers need relations across documents | Build a knowledge graph | GraphRAG |
The RAG types simulation and agentic RAG simulation compare them on the same questions; the GraphRAG simulation shows graph traversal.
RAG or fine-tuning?
Use RAG for knowledge that changes or must be cited; use fine-tuning to change style, format or behaviour. Many systems use both. The article RAG types explained covers the trade-offs in depth.
Frequently asked questions
How big should chunks be?
Commonly 200 to 800 tokens with some overlap, tuned on your own evaluation set; structure-aware chunking by headings usually beats fixed sizes.
Do long context windows make RAG unnecessary?
No. Retrieval keeps cost and latency down, scales beyond any window, and gives precise citations.
Summary
Chunk well, embed and index, retrieve with hybrid search, re-rank, generate with citations, and measure. Add query rewriting, agents or graphs only where evaluation shows the simple pipeline fails.
Back to the topic
This page is part of Generative AI and LLMs, which maps every concept, tool, simulation and article on the subject.