Naive RAG

Overview

Naive RAG is the simplest form of retrieval-augmented generation. A system splits documents into chunks, turns each chunk into an embedding, stores the embeddings in an index, and at question time embeds the question, fetches the nearest chunks and pastes them into the prompt of a language model. The model then writes an answer using that text as its source. The whole flow has three stages: index, retrieve, generate.

Naive RAG matters because it is the baseline that every other retrieval design is measured against. It is quick to build, easy to explain and often good enough for a first version. It also fails in predictable ways: poor chunk boundaries, weak matching, noisy context and no check on whether the evidence answers the question. Understanding those failures is the best way to see why Advanced RAG, Modular RAG and Agentic RAG exist.

What it is

Naive RAG is a fixed pipeline. A question goes in, passages come out of an index, and a model writes an answer. Nothing in the pipeline inspects the quality of the passages, rewrites the question or decides whether retrieval is needed. The word "naive" does not mean "bad". It means "no extra steps".

The pattern solves a real problem. A language model only knows what was in its training data, and that data has a cutoff and gaps. It also cannot see private documents. Retrieval fixes both by giving the model fresh, relevant text at the moment of the question.

The three stages

• Indexing: load documents, clean them, split them into chunks, embed each chunk and store the vectors with the original text.
• Retrieval: embed the question with the same embedding model and return the top k chunks by similarity.
• Generation: build a prompt that contains the question and the retrieved chunks, then ask the model to answer from them.

Think of a student with an open-book exam. The index is the table of contents and the highlighted notes. Retrieval is flipping to the right pages. Generation is writing the answer with the book open. If the student flips to the wrong pages, the answer will be poor no matter how well they write.

What is in the prompt

A typical naive prompt has three parts: an instruction, the retrieved context and the question.

Answer the question using only the context below.
If the context does not contain the answer, say you do not know.

Context:
[1] {chunk_1}
[2] {chunk_2}
[3] {chunk_3}

Question: {question}
Answer:

The instruction to say "I do not know" is small but important. Without it, many models will fill gaps with plausible guesses.

What it is not

• It is not fine-tuning. The model weights do not change.
• It is not a search engine. A search engine returns links, while RAG writes a new answer.
• It is not memory. The index stores documents, not a record of the conversation, unless you add that on purpose.

Where it came from

Retrieval-augmented generation was named in a 2020 paper by Lewis and colleagues. They combined a neural retriever over a text corpus with a sequence-to-sequence generator, and showed gains on open-domain question answering. Earlier work on dense passage retrieval, including the DPR paper from the same year, showed that learned embeddings could beat classic keyword matching for finding answer passages.

Before that, question answering systems used a retrieve-then-read design with keyword search such as BM25 and a separate reader model that extracted a span. RAG replaced the span reader with a generator that writes free text.

From research to everyday practice

The pattern spread quickly once large language models became available through simple interfaces. Three things made naive RAG easy to adopt:

• Embedding models that turn text into vectors with a single call.
• Vector stores and libraries such as FAISS, pgvector, Qdrant and Pinecone that make nearest-neighbour search simple.
• Orchestration libraries such as LlamaIndex that wrap loading, chunking, indexing and querying in a few lines.

The term "naive RAG" itself became popular in survey writing, where authors grouped the field into naive, advanced and modular stages. That grouping is a teaching device. Real systems blend the stages.

How it works

This section walks through a naive RAG system from raw files to a final answer, with everyday analogies.

Step 1: Load and clean documents

The system reads files such as PDFs, web pages, wiki pages or exports from a database. It strips menus, page numbers and other noise. Clean input matters more than most people expect, because the index can only be as good as the text that goes into it.

Analogy: before building a library catalogue, you remove the loose staples and torn covers.

Step 2: Split into chunks

A whole book cannot be matched to a short question, so documents are split into chunks. A common starting point is a few hundred tokens per chunk, with a small overlap between neighbours so that a sentence cut at a boundary still appears whole in one of them.

Analogy: cutting a long scroll into index cards. Cards that are too big hold many topics and blur together. Cards that are too small lose the context around a fact.

def split(text, size=400, overlap=60):
    words = text.split()
    chunks, start = [], 0
    while start < len(words):
        end = start + size
        chunks.append(" ".join(words[start:end]))
        start = end - overlap
    return chunks

Step 3: Embed each chunk

An embedding model turns each chunk into a vector, a list of numbers, so that chunks with similar meaning land close together in space. The same model must be used for questions later.

Analogy: giving every index card a map coordinate, so that cards about similar topics sit in the same neighbourhood.

Step 4: Store in an index

The vectors and their source text, plus a few metadata fields such as title and source, go into a vector store. For small collections an exact search over all vectors is fine. For larger ones, approximate indexes such as HNSW trade a little accuracy for large gains in speed.

Step 5: Embed the question

At query time, the user's question is embedded with the same model. The question vector is a point on the same map.

Step 6: Retrieve the top k

The system finds the k nearest chunks by cosine similarity or dot product. A typical k is between three and ten. The results are ranked by similarity score and returned with their text.

Analogy: standing on the map at the question's location and grabbing the k nearest index cards.

Step 7: Build the prompt

The chunks are placed into a prompt template with the instruction and the question. The total must fit inside the model's context window, with room left for the answer.

Step 8: Generate the answer

The language model reads the prompt and writes an answer. A good template asks it to cite chunk numbers so a reader can check the source.

Step 9: Return the answer

The system returns the answer, often with the source chunks or links. In the naive version, there is no further check.

The whole thing in code

def build_index(docs, embed, store):
    for doc in docs:
        for i, chunk in enumerate(split(doc.text)):
            store.add(vector=embed(chunk),
                      text=chunk,
                      meta={"source": doc.id, "chunk": i})

def answer(question, embed, store, llm, k=5):
    q_vec = embed(question)
    hits = store.search(q_vec, top_k=k)
    context = "\n".join(f"[{n+1}] {h.text}" for n, h in enumerate(hits))
    prompt = TEMPLATE.format(context=context, question=question)
    return llm(prompt), hits

The code is about a dozen lines of logic. That small size is the reason naive RAG is so widely used as a first version.

Key concepts and terms

A worked example

Imagine a small help centre with 200 short articles. A team builds a naive RAG assistant.

Setup

• Each article is split into chunks of about 300 words with 50 words of overlap.
• This yields roughly 900 chunks.
• Each chunk is embedded and stored in a vector index.
• The retriever returns the top 4 chunks.

A good case

Question: "How do I reset my password if I no longer have access to my email?"

The question embedding lands near a chunk titled "Account recovery without email". That chunk is returned first. The model reads it and writes a short answer with the steps, citing chunk 1. This is the case naive RAG is built for: one question, one clear passage.

A failing case: split answer

Question: "Which plan includes single sign-on, and how do I turn it on?"

The first half of the answer lives in a plan comparison article. The second half lives in a security settings article. The top 4 chunks are all from the settings article, because the question mentions "turn it on". The model writes the steps but does not say which plan is needed. The answer looks complete and is only half right.

A failing case: vague wording

Question: "Why was I charged twice?"

The article that covers this is titled "Duplicate payments and pending holds". The word "charged" appears but "twice" does not, and the embedding of the short question is vague. The retriever returns a chunk about invoices. The model answers about invoices. It never mentions pending holds, which was the real cause.

A failing case: missing evidence

Question: "Do you support a feature that was released last week?"

The index was built a month ago. No chunk mentions the feature. If the prompt does not insist on "I do not know", the model may guess. Even with that instruction, the retrieved chunks look related, which tempts the model to stretch them.

Reading the failures

These are the reasons the field moved to Advanced RAG with query rewriting, hybrid search and cross-encoder reranking.

A numeric view of top-k

Suppose a test set has 100 questions. For each, a person marked the chunk that holds the answer. You can measure how often that chunk appears in the top k results.

The table shows a trend, not measured values. Raising k raises recall but adds noise and cost. Finding a good k on your own data is a core tuning step.

Strengths and limitations

Strengths

• Fast to build. A working demo can exist in a day.
• Easy to understand and debug. Each stage has one job.
• Cheap to run. One embedding call and one generation call per question.
• Works with private and fresh data without retraining a model.
• Gives a natural place to attach citations, which helps trust.
• Updates are simple. Add or remove chunks and re-embed them.

Limitations

• Chunk boundaries can separate facts that belong together.
• A single dense embedding of a short question is often a weak match.
• Exact terms such as codes, names and numbers can be missed by pure vector search.
• The top k by similarity is not always the top k by usefulness.
• No check on whether the retrieved text really answers the question.
• No handling of multi-part questions.
• The model may ignore the context or mix it with its own memory.
• Long contexts can bury the key passage in the middle, where models tend to pay less attention.

Where the limits come from

Most of the weaknesses trace back to one choice: the system treats retrieval as a single, one-shot nearest neighbour lookup. Similar text is not always relevant text. A question about the cause of a problem may be close, in vector space, to a passage that only describes the problem.

How it compares with alternatives

Naive RAG versus fine-tuning

Fine-tuning changes how a model behaves. RAG changes what the model can see. If a fact changes often, retrieve it. If a style or format must be learned, tune for it. Many systems use both.

Naive RAG versus long context

Modern models accept long prompts, so it is tempting to paste an entire manual into every call. That works for small corpora. It costs more per call, it can bury the key fact, and it does not scale to large collections. Retrieval selects what matters before the model reads it.

Naive RAG versus classic search

A keyword engine returns documents and leaves the reading to the user. Naive RAG reads for the user and writes a summary. The price is the risk of an unsupported summary, which is why citations and refusal rules matter.

Role in tabular foundation models

Tabular foundation models learn from raw tables. A table has rows and columns, mixed types such as numbers, categories and text, missing values, and sometimes no useful column names. These models are usually pre-trained on large numbers of synthetic or real tables, then use in-context learning: they read labelled example rows in the prompt and predict for new rows without updating weights. TabPFN is a well-known example, and FT-Transformer is a related deep model for tabular inputs.

Naive RAG is a text pattern, not a table learner. It still relates to tabular models in concrete ways.

Retrieval chooses the context rows

In-context learning needs example rows in the context, and large tables do not fit. The retrieve-then-use idea of naive RAG transfers directly: embed or index the rows, find the nearest rows to the target and give those to the tabular model as its context. This is retrieval-augmented prediction. It is "naive" when it simply takes the top k nearest rows with no balancing or checks.

Serialising rows for a language model

If a plain language model must read a table, rows have to become text. A naive approach writes each row as a sentence or as key-value pairs, embeds it and stores it. This loses some structure. Numbers become tokens, column order may be forgotten, and missing values may look like the word "None". Care with formatting helps a lot.

customer_id=C-1042 | age_band=35-44 | plan=standard | monthly_usage=312 | tickets=2 | churned=NA

Missing values and absent column names

Retrieval by text similarity assumes the text is meaningful. If columns are named "c1", "c2" and so on, or if many values are missing, embeddings of serialised rows carry little meaning. Better options are to profile the table, infer column meaning, and fill gaps with an imputation step such as imputation networks before indexing. Related work on semantic labelling, such as semantic sector heads, tries to infer what a table is about from its columns and values alone.

Documents around the tables

Tables rarely live alone. Data dictionaries, policy pages and change logs explain them. A naive RAG index over those documents lets an assistant explain a prediction or an anomaly in words. For anomaly detection, the detector gives a score for a row and the RAG system retrieves notes that might explain it, such as a known outage or a pricing change.

Freshness without retraining

Tabular data drifts. A foundation model with in-context learning can be given recent rows as context, which is a light form of continual adaptation. A naive retrieval layer that always pulls the newest and most similar rows keeps predictions current. It does not replace retraining when the data shape changes a lot.

Honest limits

• Naive RAG does not learn from tables. It only retrieves.
• Nearest rows by text embedding are not always the best context rows for a tabular model. Distance in feature space often works better.
• Leakage is easy: if the retrieved rows include the answer for the test row, results look better than they are. Keep a clean split.
• Row-level data can be sensitive. Apply the same access rules to the index as to the source tables.

For how such models are built and measured, see /model/architecture, /model/research and /model/benchmarks.

Practical guidance and best practices

Start with a baseline, then measure

Build the naive pipeline first. Create a set of 30 to 100 real questions with known answers and known source passages. Measure retrieval recall and answer quality. Only then decide what to improve. Without a baseline, every change is a guess.

Chunking

• Split on natural boundaries such as headings and paragraphs before falling back to size limits.
• Keep a title or section path with each chunk so it is understandable alone.
• Use a small overlap, not a large one. Large overlap wastes space and returns near duplicates.
• Keep tables and code blocks whole when possible.
• Try two or three chunk sizes on your own questions. The best size depends on your data.

Embeddings

• Use the same embedding model for chunks and questions.
• Pick a model that suits your language and domain.
• Normalise vectors if your store expects it.
• Record the model name and version with the index. A change of model needs a full re-index.

Retrieval settings

• Tune k on your questions. Start with 4 to 6.
• Set a minimum similarity threshold so that clearly irrelevant chunks are dropped.
• Store metadata and use filters, for example by product, date or language.
• Remove near duplicates before building the prompt.

Prompting

• Tell the model to answer only from the context.
• Tell it what to say when the context is not enough.
• Ask for citations by chunk number.
• Put the most relevant chunks first and last if the context is long.
• Keep the instruction short and the format consistent.

Evaluation

Separate the two halves of the system. Retrieval can be good while generation is poor, and the reverse. Measure each on its own.

• Retrieval: recall at k, and whether the gold passage is in the results.
• Generation: faithfulness to the context, relevance to the question and completeness.

Tools such as RAGAS and TruLens offer metrics for these parts, and an LLM-as-judge can score faithfulness at scale. Always check a sample by hand.

Operations

• Re-index when documents change, and keep a record of what was indexed when.
• Log questions, retrieved chunks and answers for later review.
• Track latency of the embedding call, the search and the generation separately.
• Apply access control so users only retrieve what they may read.

When to move beyond naive

Move up the ladder when you can name the failure that the next step fixes.

A debugging checklist for bad answers

When an answer is wrong, walk through the stages in order. Each check is quick and points to a specific fix.

1. Was the source document indexed at all? If not, fix loading or the file filter.
2. Was the text extracted cleanly? Open the stored chunk and read it. Broken tables and garbled columns are common.
3. Does the gold passage sit inside a single chunk, or was it cut in two? If cut, change the splitting rule.
4. Does the gold chunk appear in the top 20 results, even if not the top 5? If yes, reranking will help. If no, the query and chunk wording are too far apart.
5. Is the prompt too long? Count tokens. If the key chunk is in the middle of a very long context, reduce k.
6. Did the model follow the instruction to use only the context? If not, tighten the template and lower the temperature.
7. Is the answer correct but unsupported by the cited chunk? Add a faithfulness check.

Troubleshooting table

A small evaluation configuration

Keeping the settings of each experiment in a file makes comparisons fair.

experiment: baseline-v1
chunking:
  size_tokens: 300
  overlap_tokens: 50
embedding_model: example-embedder-v1
retrieval:
  top_k: 5
  min_score: 0.25
generation:
  model: example-llm
  temperature: 0.1
metrics: [recall_at_5, faithfulness, answer_relevance]
questions_file: eval/questions.jsonl

Cost and latency in a naive pipeline

A naive pipeline has two model calls per question: one embedding call and one generation call. Generation dominates both time and money, and it grows with the length of the prompt. Cutting k from ten to five roughly halves the retrieved text, which cuts the prompt cost with little loss if recall is stable. Caching the embedding of repeated questions and the answers to frequent ones removes many calls altogether.

Keeping the index healthy

• Store a content hash for each chunk so unchanged documents are not re-embedded.
• Delete chunks when their source is removed, so old answers do not linger.
• Version the index. Keep the previous version until the new one passes the test set.
• Schedule a small regression run after every re-index.

Common mistakes

• Skipping evaluation and judging the system by a few demo questions.
• Using chunks that are far too large, so each one mixes many topics.
• Using chunks that are far too small, so facts lose their context.
• Embedding questions and chunks with different models.
• Forgetting to re-index after documents change.
• Setting k very high in the hope that more context is always better.
• Not telling the model what to do when the context lacks the answer.
• Letting the model answer from memory and presenting it as if it came from the documents.
• Ignoring exact-match needs such as product codes, error numbers and names.
• Indexing duplicated or outdated versions of the same document.
• Dropping metadata, which makes filtering and citations impossible.
• Treating retrieved text as trusted. A document can contain instructions meant to steer the model. See prompt injection defense.
• Skipping access control, so a user can read chunks from documents they should not see.
• Measuring only the final answer and never checking what was retrieved.

Frequently asked questions

What is naive RAG?

Naive RAG is the basic retrieval-augmented generation pattern. It splits documents into chunks, embeds them, finds the chunks nearest to a question and gives them to a language model as context for the answer. There are no extra steps for rewriting, reranking or checking.

Why is it called naive?

The name marks the simplicity of the design, not a flaw in the idea. It does a single retrieval pass with no query improvement, no reranking and no verification. Survey writing uses the label to contrast it with advanced and modular designs.

Is naive RAG good enough for production?

Sometimes. For a small, clean collection with simple questions it can work well and is cheap to run. For large, messy or varied collections it usually needs better chunking, hybrid search and reranking. Measure on your own questions before deciding.

How big should my chunks be?

There is no single right size. A few hundred tokens is a common starting point, with a small overlap. Smaller chunks give more precise matches but lose context. Larger chunks keep context but blur topics. Test two or three sizes against a set of real questions.

How many chunks should I retrieve?

Start with about four to six and tune from there. Too few chunks risk missing the answer. Too many add noise, raise cost and can hide the key passage. A similarity threshold helps drop chunks that are clearly unrelated.

Does naive RAG stop hallucinations?

It reduces them but does not remove them. The model can still ignore the context, mix it with memory or stretch weak evidence. Clear instructions, citations, a refusal rule and a faithfulness check lower the risk.

What is the difference between naive RAG and advanced RAG?

Advanced RAG keeps the same basic flow and adds steps before and after retrieval. Before: query rewriting, expansion or routing. After: reranking, filtering and compression. These steps raise the precision and recall of the evidence the model sees.

Can naive RAG work with tables and spreadsheets?

Yes, with care. Rows can be serialised into text and indexed, but structure and numbers suffer. For numeric questions, a SQL tool or a tabular model is often better, and the RAG layer can supply the surrounding documents and explanations.

Do I need a vector database?

Not at first. A small collection can be searched with a simple in-memory index or a library. A dedicated store becomes useful when the collection grows, when you need filters and updates, or when several services share the index.

How do I know if retrieval or generation is at fault?

Look at the retrieved chunks for a bad answer. If the needed passage is absent, retrieval failed. If it is present and the answer is still wrong, generation failed. Logging both makes this check quick.

Glossary

• Approximate nearest neighbour search: a fast method that finds close vectors without checking every one.
• Chunk: a piece of a document stored and retrieved as a unit.
• Context window: the amount of text a model can read at once.
• Cosine similarity: a measure of how closely two vectors point in the same direction.
• Embedding: a numeric vector that represents the meaning of text.
• Faithfulness: how well an answer sticks to the supplied evidence.
• Grounding: tying an answer to retrieved evidence.
• Hallucination: an unsupported or false claim written fluently.
• Index: the stored structure used to search chunks quickly.
• In-context learning: learning from examples shown in the prompt, with no weight updates.
• Metadata filter: a rule that limits search to chunks with certain field values.
• Overlap: repeated text at the edges of neighbouring chunks.
• Recall at k: the share of questions whose gold passage appears in the top k results.
• Retriever: the component that finds relevant chunks.
• Top-k: the number of chunks returned from a search.

Summary

Naive RAG embeds a question, fetches the nearest document chunks and passes them to a language model as context. It is simple, cheap and a strong first version, and it gives every other retrieval design a baseline to beat.

Its weaknesses are also clear: poor chunk boundaries, vague query matching, noisy context and no check on support. Measure retrieval and generation separately, tune chunk size and k on your own questions, and add steps such as rewriting, hybrid search and reranking only when a named failure calls for them. For tabular work, treat the pattern as a way to fetch context rows and explanatory documents for a tabular foundation model. Continue with Advanced RAG, hybrid search, GraphRAG and /model/research.

Related

It belongs to RAG in my stack, under LLMs, retrieval, and evaluation. Others in the same group: