Contextual Retrieval

Overview

Contextual Retrieval is a technique for retrieval-augmented generation (RAG) that fixes a common weakness of chunking: a chunk, cut out of its document, often loses the information that makes it understandable. The method asks a language model to write a short piece of context for each chunk, such as which document it comes from, which section it belongs to and what it is about. That context is placed in front of the chunk text before the chunk is embedded and before it is added to a keyword index. The result is better recall, because the chunk now carries the words and meaning that its original surroundings supplied.

The idea is simple, cheap to try and works with almost any retrieval stack. It is applied once, at indexing time, so query-time latency does not grow. It pairs well with hybrid search (the context helps both the dense and the BM25 side) and with reranking. For table-centric systems, the same idea means prefixing a row, a column description or a table fragment with the table name, the schema and the meaning of the fields, so that small pieces of tabular data remain understandable on their own.

What it is

Contextual Retrieval is an indexing-time enrichment step. Instead of indexing a raw chunk such as "Revenue grew 3% over the previous quarter", you index a chunk with a one or two sentence lead-in such as "This chunk is from the quarterly report of a fictional retail company for the second quarter. Revenue grew 3% over the previous quarter." The chunk is now clear about whose revenue and which quarter, even when it is retrieved alone.

It has two parts that are usually used together.

• Contextual embeddings. The context plus the chunk is sent to the embedding model, so the vector reflects both.
• Contextual keyword index. The context plus the chunk is also added to a lexical index such as BM25, so exact terms from the context can match query words.

The technique is separate from the choice of embedding model, vector store or generator. It is a preprocessing layer. You can think of it as writing a good label on every page that you tear out of a book.

An everyday analogy

Imagine a filing room where each page has been pulled from its folder and put in a big pile. A page that says "The deadline is moved to Friday" is useless on its own. Which project? Which Friday? Now imagine a clerk who clips a small note to each page: "Project Alpha, kickoff memo, March". Searching the pile becomes easy. Contextual Retrieval hires that clerk, and the clerk is a language model.

What the context contains

Good context is short and factual. It usually has these elements.

• The document title or type.
• The section or heading path.
• The entity the chunk is about, such as a company, a product or a table.
• The time frame, version or scope if relevant.
• One phrase that resolves pronouns or vague references, such as "the company" or "this change".

Context should not repeat the chunk, add claims that are not in the source, or grow long. A typical length is a sentence or two.

Where it came from

Contextual Retrieval grew out of practical experience with RAG systems. Early RAG pipelines split documents into fixed-size chunks and embedded each one alone. Teams noticed that many retrieval failures came from chunks that had lost their antecedents: pronouns without referents, figures without units, and sections without titles.

Earlier practices tried to ease the problem in simpler ways.

• Adding the document title or heading path as a prefix to each chunk.
• Overlapping chunks so that neighboring text is shared.
• Parent-document retrieval, where a small chunk is matched but a larger parent block is returned.
• Summaries stored with each chunk, or document-level summaries added to every chunk.

The 2024 formulation named and popularized the approach of generating the context with a language model, per chunk, using the full document as reference, and applying it to both embeddings and keyword indexes. Its practical appeal came from two things: a language model can write better situating text than a rule-based prefix, and caching of the long document prompt makes the extra generation affordable. Reported gains in public write-ups were meaningful, especially when combined with a keyword index and a reranker, but the size of the gain depends on the corpus, so measure it on your own data rather than trust a general figure.

It sits in the family of ideas described on advanced RAG and modular RAG pages: improve what is indexed, improve the query, then improve the ranking.

How it works

The method has an indexing phase and a query phase. Only the indexing phase changes.

Step 1: Split documents into chunks

Split each document as usual. Use headings and paragraphs where possible, and pick a size that fits your embedding model. Typical sizes are a few hundred tokens. Keep a record of the document ID and the chunk position.

Step 2: Prompt a language model for context

For each chunk, give the model two things: the whole document (or a large enough window around the chunk) and the chunk itself. Ask for a short context that situates the chunk within the document for the purpose of search. The prompt is usually something like the following.

<document>
{{WHOLE_DOCUMENT}}
</document>

Here is the chunk we want to situate within the whole document:
<chunk>
{{CHUNK_CONTENT}}
</chunk>

Write a short context (one or two sentences) that places this chunk
within the overall document, to improve search retrieval of the chunk.
Answer with the context only.

The output is short, so the cost of output tokens is small. The input is large, because the whole document is repeated for each chunk. That is the main cost, and it is where prompt caching helps.

Step 3: Prepend the context to the chunk

Join the generated context and the original chunk into one string. Many teams store the original chunk separately too, so the generator can be shown the clean text, or the combined text, depending on the experiment.

Step 4: Embed the combined text

Send the combined string to the embedding model. Store the vector with the chunk ID. Any vector index or library such as FAISS, or a database extension such as pgvector, can hold it.

Step 5: Index the combined text for keywords

Add the same combined string to a lexical index, for example BM25 or a search engine such as Elasticsearch. Now a query word that appears only in the context, such as a company name or a product code, can match.

Step 6: Retrieve and fuse

At query time, run both retrievers. Fuse the ranked lists, for example with reciprocal rank fusion, as described on the hybrid search page. Optionally rerank the top candidates with a cross-encoder or a late-interaction model such as ColBERT.

Step 7: Generate with citations

Send the top chunks to the generator. Show the original chunk text, and keep the context only as metadata, or show both. Ask the generator to cite chunk IDs.

The pipeline in one view

documents
   |
   v
split into chunks
   |
   v
for each chunk: LLM writes context (document + chunk -> 1-2 sentences)
   |
   v
context + chunk
   |            \
   v             v
embedding index   keyword index (BM25)
   \             /
    v           v
   query -> both retrievers -> fuse -> rerank -> generator

Why it works

Embeddings average the meaning of the words they see. A bare chunk with vague words yields a vague vector, which sits close to many other vague vectors. The context adds distinguishing words, so the vector moves toward the right neighborhood. The same logic holds for keyword matching: a keyword index can only match words that are present, so adding the missing entity name or topic gives the index something to match.

In other words, the technique raises the information content of each stored unit. It does not make the retriever smarter. It makes the thing being retrieved easier to find.

Prompt caching and cost

The expensive part is repeating the document for every chunk. If the model provider supports caching of a repeated prompt prefix, the document is processed once and re-read cheaply for each chunk. Structure the prompt so the document comes first and the chunk comes last, since caching works on a shared prefix. Even without caching, indexing is a one-time cost per document version, and it is often small compared with the cost of serving queries over time.

Cost grows with three factors: the number of chunks, the length of the document context and the model used. Use a small, fast model for context writing, since the task is simple.

Handling very long documents

If the document is too long to fit in the model's window, give it a window instead: the section that holds the chunk, the table of contents, and a short summary of the whole document. Another option is to build a hierarchy: summarize each section, then give the section summary and the chunk to the model.

Key concepts and terms

A worked example

Take a small corpus: two fictional quarterly reports from two fictional companies, Northwind Foods and Contoso Tools. Each is split into chunks. Consider these two chunks.

• Chunk A, from Northwind: "Revenue grew 3% over the previous quarter, driven by the dairy segment."
• Chunk B, from Contoso: "Revenue grew 3% over the previous quarter, driven by the hardware segment."

The query is: "How much did Northwind Foods revenue grow in Q2?"

Without context, both chunks are nearly identical to the embedding model, and neither mentions Northwind or Q2. The retriever may return either, or return one from the wrong company. With context, the chunks become:

• A: "From the Northwind Foods report for the second quarter. Revenue grew 3% over the previous quarter, driven by the dairy segment."
• B: "From the Contoso Tools report for the second quarter. Revenue grew 3% over the previous quarter, driven by the hardware segment."

Now the keyword index matches "Northwind" and "Foods" in A, and the embedding of A leans toward the Northwind topic. The right chunk rises to the top.

Code: generating context for each chunk

The sketch below uses a placeholder function call_llm that you replace with your own model client. Hosts and names are fake.

from dataclasses import dataclass

@dataclass
class Chunk:
    doc_id: str
    pos: int
    text: str

PROMPT = """<document>
{doc}
</document>

Here is the chunk we want to situate within the whole document:
<chunk>
{chunk}
</chunk>

Write a short context (one or two sentences) that places this chunk
within the overall document, to improve search retrieval of the chunk.
Answer with the context only."""

def call_llm(prompt: str) -> str:
    # Replace with a real client call, for example to
    # https://llm.example.internal/v1/complete
    raise NotImplementedError

def contextualize(doc_text: str, chunks: list[Chunk]) -> list[dict]:
    out = []
    for c in chunks:
        ctx = call_llm(PROMPT.format(doc=doc_text, chunk=c.text)).strip()
        out.append({
            "doc_id": c.doc_id,
            "pos": c.pos,
            "context": ctx,
            "original": c.text,
            "combined": f"{ctx}\n\n{c.text}",
        })
    return out

Code: indexing both ways and fusing results

import math
from collections import defaultdict

def rrf(rank_lists, k=60):
    """Reciprocal rank fusion over lists of chunk IDs."""
    score = defaultdict(float)
    for ranking in rank_lists:
        for rank, cid in enumerate(ranking, start=1):
            score[cid] += 1.0 / (k + rank)
    return sorted(score, key=score.get, reverse=True)

def search(query, dense_index, bm25_index, top_n=150, final_k=20):
    dense_ids = dense_index.search(query, top_n)    # returns chunk IDs
    bm25_ids = bm25_index.search(query, top_n)      # returns chunk IDs
    fused = rrf([dense_ids, bm25_ids])
    return fused[:final_k]

The text added to both indexes is the combined field. The generator can be shown the original field, or the combined one.

Measuring the gain

Create a small test set of questions, each with the chunk (or chunks) that holds the answer. Then compute recall at 5, 10 and 20 for four setups.

Compare the failed retrieval rate across rows. The pattern to expect is that each step helps a little, and that the gain is largest on corpora where chunks are short, vague or full of pronouns. Do not assume a number. Measure it.

Reading the results

If contextual embeddings help but contextual BM25 adds little, your queries are mostly about meaning. If BM25 gains most, your queries contain names and codes. If nothing helps, check whether the context is too generic. A context like "This is part of a document" adds no information.

Strengths and limitations

Strengths

• Raises recall at the source. The chunk becomes easier to find for any retriever.
• No query-time cost. The work is done at indexing time.
• Model-agnostic. It works with any embedding model, any vector store and any keyword engine.
• Complements other methods. It stacks with hybrid search, reranking, query rewriting and parent-document retrieval.
• Fixes a real failure mode. Pronouns, missing entities and missing time frames are common in real corpora.
• Easy to explain and audit. Each context is plain text that a person can read and correct.

Limitations

• Indexing cost. Every chunk needs a model call. Large corpora cost more and take longer to process.
• Risk of wrong context. A model may add an unsupported claim. A false context is worse than none, since it is indexed as if it were true.
• Re-index on change. If a document changes, the contexts of its chunks may need regeneration.
• Not a fix for bad chunking. If chunks split tables or lists mid-way, context will not repair that.
• Diminishing returns on clean, self-contained chunks. If every chunk already names its subject, the gain is small.
• Index size. Contexts add tokens to the keyword index and to the text that is embedded.
• Leakage across scope. If the model sees the whole document, the context may reveal details from other sections. Apply access rules at the document level.

How it compares with alternatives

Contextual Retrieval and query-side methods are partners rather than rivals. One improves the documents, the other improves the question. Using both closes the gap from two sides.

How to choose

• If chunks are short and vague, start with Contextual Retrieval.
• If queries are short and vague, start with query rewriting or HyDE.
• If you have both exact codes and natural language, use hybrid search.
• If you retrieve well but order poorly, add a reranker.
• If the questions need links across documents, consider a graph approach.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows and columns, mixed numeric, categorical and text cells, missing values, and often no reliable column names. Contextual Retrieval is not a part of those models. It is a way to make the surrounding retrieval layer work better when tabular material has to be found, selected and shown to a model or to a person. The connection is concrete in several places.

Tables lose context when cut apart

A row such as 42 | 7.5 | N | 2019 means nothing alone. A chunk of a CSV that starts mid-file has no header. This is the tabular form of antecedent loss. The remedy is the same as for text: attach context. For a table fragment, a good context states the table name, the meaning of each column, the units, the grain of a row (one row per order, one row per day) and the source system. Then a retriever can match the fragment to a question like "daily temperature readings in Celsius".

No column names

Some tables have anonymous columns (c1, c2, c3). Tabular foundation models are designed to cope with this, because they learn from the values and their patterns rather than from the names. A retrieval layer around them can use generated context to describe what each column seems to hold, based on value ranges, formats and neighbors. This must be marked as inferred, not as fact, and checked by a person when it matters.

Selecting rows for in-context learning

Models in the family of TabPFN learn in context: you give them a labeled table and they predict for new rows without gradient updates. When the labeled table is too big for the model's context, you must pick which rows to show. Retrieval of similar rows is a common way to do it. Contextual text for rows (for example a short natural-language rendering of a row with its field meanings) can make that similarity search more reliable, especially for mixed text and numeric rows. The final prediction still comes from the numeric reasoning of the model. Retrieval only chooses the support set. See /model/architecture for how in-context prediction works.

Imputation

To fill a missing cell, the best evidence is often in similar rows or in a related table. A retriever with contextual descriptions of tables and columns can find donor rows and the right reference table. Imputation methods such as imputation networks and masked cell modeling then use that evidence, or the model fills the cell by itself. Context helps decide which sources are valid for a given column, for example that "amount" in one table is in cents and in another is in dollars.

Anomaly detection

For anomaly detection, retrieval of nearest neighbors gives a local picture of what is normal. Context that explains the grain and units reduces false alarms, since a value that looks extreme in cents may be normal in dollars. Retrieved descriptions of past incidents, stored as contextualized chunks, can also help an analyst understand why a flagged row matters.

Sector and semantic classification

Classifying a table by domain or a column by meaning, as semantic sector heads do, benefits from reference material: taxonomy entries, glossaries and examples of columns per sector. These reference chunks are short and easily ambiguous ("rate", "status", "type"). Contextual Retrieval makes them findable by attaching the taxonomy path and a short definition. A column called "rate" under a finance taxonomy path is then distinct from "rate" under a healthcare path.

Continual learning

Tabular models and their surrounding knowledge change. Retrieval offers a way to add new knowledge without retraining: new documents, new schemas and new taxonomy entries are contextualized and added to the index. This is a complement to training-based continual learning such as experience replay, not a substitute. Weights still hold what the model learned from data. The index holds facts that change often.

Honest limits

Generated context about a table is a hypothesis. A language model may guess the meaning of a column wrongly. Treat contexts as metadata with a confidence level, keep them separate from the numeric content, and never let them overwrite the values in the table. Also note that Contextual Retrieval does not teach a model to reason about numbers. It only improves what gets found. For research and measured results on the model side, see /model/research and /model/benchmarks.

Practical guidance and best practices

Start with a baseline and a test set

Before changing anything, build a small set of real questions with known answer chunks. Measure recall at several depths. Without this, you cannot tell if the technique helped.

Write a strong context prompt

• State the purpose: "to improve search retrieval".
• Ask for one or two sentences.
• Ask for the context only, with no preamble.
• Tell the model not to add facts that are not in the document.
• Ask it to name entities, time frames and versions where they exist.

Pick a small, fast model

Context writing is a short, simple task. A small model is often enough. Check a sample by hand, then decide.

Use caching

Put the document first and the chunk last in the prompt. Reuse the document prefix for all chunks of that document. This cuts both cost and time.

Combine with hybrid search and reranking

The strongest results in public write-ups came from using contextual embeddings and contextual keyword indexes together, then reranking. Do the first two before adding a reranker, since they make the candidate pool better.

Keep original and context separate in storage

Store three fields: the original text, the context, and the combined text. This lets you change what is embedded, what is shown and what is cited without recomputing everything.

Version your contexts

Record the prompt version, model name and date. If the prompt changes, you know which chunks were built with which version.

Handle tables and code carefully

Keep a table whole where possible, with its header repeated for each fragment. For code, include the file path and the function or class name in the context.

Review a sample by hand

Read fifty random contexts. Look for vague text, wrong claims and repeated boilerplate. Fix the prompt, not individual contexts.

Limit length

Long contexts dilute the chunk. Keep them under a few dozen words where you can.

Set up a refresh policy

When a document changes, regenerate contexts for its chunks. When only a distant section changes, you may be able to keep the old contexts, but check that no context refers to the changed part.

Config sketch

contextual_retrieval:
  chunking:
    max_tokens: 400
    overlap_tokens: 40
    keep_tables_whole: true
  context:
    model: small-fast-model
    max_words: 50
    prompt_version: v3
    cache_document_prefix: true
  index:
    dense:
      text_field: combined
      store: vector-store.example.internal
    lexical:
      text_field: combined
      store: search.example.internal
  retrieval:
    top_n_each: 150
    fusion: rrf
    rerank_top: 150
    final_k: 20

Common mistakes

• Generic contexts. "This is a chunk from a document" adds nothing. Require named entities and scope.
• Letting the model invent facts. Add an instruction to use only the document, and spot check.
• Skipping the keyword index. Contextual embeddings alone leave gains on the table; the keyword side often benefits a lot.
• Embedding the context but showing only the context. The generator should see the real source text.
• Ignoring caching. Paying full price to re-read the document for every chunk makes indexing needlessly slow.
• No test set. Gains and losses go unnoticed.
• Over-long contexts. They drown the chunk and raise cost.
• Poor chunking. Splitting in the middle of a table or sentence cannot be repaired by context.
• Forgetting to refresh. Old contexts that mention a previous version mislead retrieval.
• Treating contexts as truth. They are generated text. Keep provenance and allow correction.
• Mixing access levels. A context built from a restricted document must carry the same restriction.

Frequently asked questions

What is Contextual Retrieval in simple terms?

Contextual Retrieval means adding a short, generated note to each chunk of a document before you index it, so the chunk still makes sense when it is found alone. The note says what the chunk is about and where it comes from. Both the embedding and the keyword index use the chunk together with its note, which makes the right chunk easier to retrieve.

How is it different from adding a document title to each chunk?

A title prefix is a fixed rule and gives the same label to every chunk of a document. Contextual Retrieval writes a different note for each chunk, so it can name the specific section, entity, time frame and meaning of that piece. A title is still a good cheap baseline, and it is worth testing first, but generated context usually carries more distinguishing detail.

Does it make queries slower?

No. The extra work happens when documents are indexed, not when questions are answered. At query time the system searches the same kinds of indexes as before. The only change is that the stored text is richer. If you add a reranker at the same time, that part adds latency, but the contextual step itself does not.

Is it expensive to run?

It adds one model call per chunk, with the document as input, so cost depends on corpus size and document length. Using a small model and caching the repeated document prefix keeps it modest. Because it is a one-time cost per document version, many teams find it small compared with the ongoing cost of serving queries.

Should I use it with BM25 as well as embeddings?

Yes, in most cases. The context often contains names, codes and terms that a keyword index can match exactly, while the embedding captures the overall meaning. Using the contextualized text in both indexes and fusing the results, as in hybrid search, tends to work better than using either alone. Test both on your own questions.

Can the generated context be wrong?

Yes. A language model can misread a document or add a claim that is not there. Because the context is indexed as if it were part of the source, an error can mislead retrieval. Reduce the risk with a clear instruction to use only the document, a sample review by a person, and a habit of showing the original chunk to the answer generator.

Does it replace reranking or query rewriting?

No. It improves the stored chunks, while reranking improves the final order and query rewriting improves the question. They solve different problems and work well together. A common order of adoption is hybrid search first, then contextual chunks, then a reranker, and query-side methods when users ask vague questions.

How does it apply to tables and spreadsheets?

A table fragment loses its header, units and grain when cut out. Contextual Retrieval fixes this by attaching the table name, the meaning of the columns, the unit of each field and what one row stands for. That makes fragments findable by meaning, and it helps systems that choose rows or schemas for a tabular model to work with. The values themselves must stay unchanged.

When is it not worth using?

If your chunks already name their subject and scope, or if the whole corpus is small enough to fit in the model's context, the gain may be small. It is also less useful when the main failure is poor chunk boundaries or a weak embedding model. Measure on a test set first, and use the technique where the data shows a retrieval gap.

How do I know it worked?

Compare recall at several depths, or the failed retrieval rate, between a baseline index and the contextual index on the same questions. Also check answer quality end to end with a rubric or a judge model, because better recall should lead to better answers. If recall improves but answers do not, look at the generator prompt and the context size.

Glossary

• Antecedent loss: The loss of what a word like "it" or "the company" refers to when text is cut out of its document.
• BM25: A classic keyword ranking function based on term frequency and rarity.
• Cache (prompt): Stored processing of a repeated prompt prefix that makes later calls cheaper and faster.
• Chunk: A piece of a document stored and retrieved as one unit.
• Combined text: The context joined to the original chunk, used for indexing.
• Context (situating): A short note that places a chunk in its document.
• Dense retrieval: Retrieval by vector similarity between query and chunk embeddings.
• Embedding: A list of numbers that represents the meaning of a piece of text.
• Failed retrieval rate: The share of questions where the right chunk is not in the top results.
• Fusion: Merging ranked lists from several retrievers into one.
• Grain (of a table): What one row stands for, such as one order or one day.
• Index-time enrichment: Improving data before it is stored, so searches work better later.
• Lexical retrieval: Retrieval by matching words.
• Parent-document retrieval: Matching a small chunk but returning its larger parent block.
• Provenance: A record of where a piece of text or a claim came from.
• Recall at k: The share of questions for which a right answer appears in the top k results.
• Reranker: A model that re-scores a short list of candidates for a better final order.
• Reciprocal rank fusion: A fusion method that sums inverse ranks from each list.
• Support set: The labeled rows shown to a model that learns in context.

Summary

Contextual Retrieval adds a short, model-written context to each chunk before it is embedded and before it is added to a keyword index. This restores the meaning that chunking removes, raises recall, and costs only at indexing time. It works best with hybrid search and reranking, needs caching to stay affordable, and must be checked for wrong or generic contexts.

For tabular foundation models and table-centric systems, the same idea keeps small pieces of tabular data understandable: table names, column meanings, units and row grain travel with each fragment. That helps retrieval of support rows for in-context learning, donors for imputation, neighbors for anomaly checks and taxonomy entries for sector classification, while the numeric reasoning stays with the models. Measure on your own questions, and see /model/research, /model/architecture and /model/benchmarks for the model side.

Related

It belongs to RAG in my stack, under LLMs, retrieval, and evaluation. Others in the same group: