HyDE

Overview

HyDE, short for Hypothetical Document Embeddings, is a retrieval technique that searches with a made-up answer instead of the raw question. A language model first writes a short passage that could plausibly answer the query. That passage may contain errors, but it has the style, vocabulary and shape of a real answer. An embedding model turns the passage into a vector, and the system looks up real documents whose vectors are close to it. The real documents are then used as evidence. The invented text is thrown away.

The method helps when queries are short, vague or phrased very differently from the documents. Questions and answers live in different parts of language space: a question says "why does my pipeline stall at night", while a document says "scheduled jobs compete for a shared lock between 01:00 and 03:00". A hypothetical answer sits much closer to such a document than the question does. HyDE needs no labeled training pairs, works with off-the-shelf models, and can be used with any dense index such as FAISS or pgvector. Its costs are an extra model call per query and a risk that the model's guess pulls the search in a wrong direction.

What it is

HyDE is a query-time technique for dense retrieval. It replaces the query embedding with the embedding of a generated pseudo-document. It is a form of query expansion, in the same family as query rewriting, but with a specific twist: the expansion is a whole passage written in the style of the corpus, and the retrieval step uses only its vector.

Two ideas sit behind it.

• Generation can bridge a vocabulary gap. A language model has read many answers and can write one that looks right, even if the facts are wrong.
• An embedding model acts as a filter for mistakes. The embedding step compresses the passage into a point in meaning space, and details that are specific and wrong tend to wash out, while the overall topic and style remain.

An everyday analogy

Suppose you want to find a certain book in a huge library but you only remember a vague question: "how did people keep food from spoiling before fridges?" Searching by that sentence turns up little. Now suppose you first jot a few lines of what an answer might say: "People used salt, smoke, drying, pickling and cellars to slow spoilage." You take your note to the shelves and look for books that read like it. Even if some of your note is wrong, it points at the right shelf. HyDE does exactly that, with a language model as the note writer and an embedding model as the shelf finder.

What it is not

• It is not an answer generator. The hypothetical passage is never shown as a fact.
• It is not a training method. No model weights are changed.
• It is not a replacement for good chunking, hybrid search or reranking. It is one tool in a retrieval stack.

Where it came from

HyDE was introduced in a 2022 research paper titled "Precise Zero-Shot Dense Retrieval without Relevance Labels". The authors wanted dense retrieval to work in new domains where no labeled query-document pairs exist. Dense retrievers are usually trained on such pairs, and they can fall short when moved to a different subject or language style.

The paper's proposal was to split the problem. Let an instruction-following language model handle the hard job of understanding the query and writing something relevant, and let an unsupervised contrastively trained encoder handle the job of grounding that text in the real corpus. The work reported strong results against unsupervised baselines and competitive results against some fine-tuned retrievers across several search tasks, though exact numbers depend on the benchmark and models, so this article does not repeat them.

The idea has several relatives.

• Classic query expansion adds related terms to a query, using thesauri or pseudo-relevance feedback.
• Doc2query goes the other way: it generates likely queries for each document at index time.
• Pseudo-relevance feedback assumes that top results are relevant and uses them to refine the query.
• Multi-query and fusion approaches generate several query variants and merge their results.
• Step-back prompting and related methods rewrite a question into a more general one.

Since then HyDE has become a standard building block in RAG toolkits. It appears among the options in frameworks such as LlamaIndex, and it is often listed among pre-retrieval methods on pages about advanced RAG and modular RAG.

How it works

Step 1: Take the user query

The input is a natural-language question or instruction, for example "How do I handle missing values in a small table?"

Step 2: Prompt a language model to write a hypothetical document

The prompt asks for a passage that answers the question, in the style of the target corpus. The style matters. If the corpus is technical documentation, ask for a documentation paragraph. If it is a set of support tickets, ask for a ticket resolution.

Write a short passage (about 100 words) that answers the question below,
in the style of internal technical documentation. It is fine if some
details are uncertain. Do not mention that you are guessing.

Question: {{QUERY}}

Passage:

Step 3: Optionally sample several passages

Language models are random. One passage may drift off topic. A common practice is to generate several passages (for example three to eight), embed each, and average the vectors. Some implementations also include the original query's vector in the average. Averaging smooths out the errors of any single guess, like asking five people to describe the same thing and keeping what they agree on.

Step 4: Embed the passage or passages

Use the same embedding model that built the document index. The vector must live in the same space as the stored vectors, or the distances mean nothing.

Step 5: Search the index

Run a nearest-neighbor search with the HyDE vector. Use an approximate index such as HNSW for speed on large collections.

Step 6: Return real documents

The hypothetical passage is discarded. The retrieved real chunks go forward, optionally after reranking, to the generator.

The pipeline in one view

query
  |
  v
LLM writes hypothetical answer(s)
  |
  v
embedding model -> vector (average of several if sampled)
  |
  v
nearest-neighbor search over real document vectors
  |
  v
real chunks -> (rerank) -> generator -> final answer with citations

Why it works

A contrastively trained encoder maps text to vectors so that texts with similar meaning land close together. But a short question and a long answer are different kinds of text. The question has few words and may use different terms. The answer has many words, specific terms and the usual shape of explanation. A document-like embedding for the query compares like with like. This is sometimes described as moving from query-to-document matching to document-to-document matching.

Why mistakes do not ruin it

The hypothetical text may contain wrong numbers or invented names. But the encoder produces a dense summary of the whole passage. Rare, specific errors have limited influence on the overall vector, while the general topic, vocabulary and structure dominate. The real documents then supply the correct facts. This works as long as the model's guess is roughly on topic. If the model knows nothing about the subject, the guess is off topic and HyDE can do worse than plain search.

Variants

Key concepts and terms

A worked example

Take a small knowledge base about data pipelines, with these three fictional chunks.

1. "Nightly batch jobs acquire a shared lock on the staging schema. When the export job overruns, later jobs wait until 03:00."
2. "The dashboard refresh uses cached aggregates. Cache keys include the tenant ID and the date."
3. "To rotate service credentials, update the secret store and restart the workers in rolling order."

The query is: "why does my pipeline stall at night?"

Plain dense search on the query may rank chunk 2 first, because "dashboard", "refresh" and "cache" feel closer to a generic pipeline discussion, or may rank them all close together. The word "stall" and "night" do not appear in chunk 1, which uses "wait", "lock" and "nightly".

With HyDE, the language model writes something like: "A pipeline can stall at night when scheduled jobs compete for the same lock or resource. If an earlier job runs long, later jobs wait in a queue until it finishes. Check the schedule, look for long-running tasks and review resource limits."

That passage has the words "scheduled", "lock", "wait", "queue" and "long-running". Its embedding sits close to chunk 1. The search now returns chunk 1 first, and the generator can quote the real details (the shared lock, the 03:00 time) from the real document.

Code: a minimal HyDE retriever

The code below is a sketch. generate and embed are placeholders for your own clients, and the index object is any vector index with a search method.

import numpy as np

HYDE_PROMPT = """Write a short passage (about 100 words) that answers the
question below, in the style of internal technical documentation.
It is fine if some details are uncertain.

Question: {query}

Passage:"""

def generate(prompt: str, temperature: float = 0.7) -> str:
    # Replace with a real call, for example to https://llm.example.internal
    raise NotImplementedError

def embed(texts: list[str]) -> np.ndarray:
    # Replace with a real embedding client; returns shape (n, d)
    raise NotImplementedError

def hyde_vector(query: str, n_samples: int = 4, include_query: bool = True):
    passages = [
        generate(HYDE_PROMPT.format(query=query), temperature=0.7)
        for _ in range(n_samples)
    ]
    vecs = embed(passages)
    if include_query:
        vecs = np.vstack([vecs, embed([query])])
    v = vecs.mean(axis=0)
    return v / np.linalg.norm(v)

def hyde_search(query: str, index, k: int = 10):
    v = hyde_vector(query)
    return index.search(v, k)   # returns list of (chunk_id, score)

Code: a hybrid use with keyword search

def hyde_hybrid(query, dense_index, bm25_index, k=20):
    dense_hits = hyde_search(query, dense_index, k=100)
    bm25_hits = bm25_index.search(query, 100)   # original query for exact terms
    return rrf([[h[0] for h in dense_hits], [h[0] for h in bm25_hits]])[:k]

def rrf(rank_lists, c=60):
    score = {}
    for lst in rank_lists:
        for r, cid in enumerate(lst, start=1):
            score[cid] = score.get(cid, 0.0) + 1.0 / (c + r)
    return sorted(score, key=score.get, reverse=True)

This keeps the original query for the keyword side, where exact words matter, and uses the passage only on the dense side. See hybrid search and BM25 for the fusion idea.

Evaluating the effect

Build a set of real queries with labeled right chunks, then compare three setups on recall at 5, 10 and 20.

Break results down by query type: short keyword-like, long natural, ambiguous, and exact-code queries. HyDE usually helps most on short and vague natural-language queries, and may hurt on queries that contain exact identifiers, because the generated passage invents identifiers or drops them.

Strengths and limitations

Strengths

• No labeled data needed. It works in new domains with general models.
• Bridges the question-answer gap. It compares document-like text with documents.
• Easy to add. It is a query-time wrapper, with no change to the index.
• Works with any dense index. The stored vectors stay as they are.
• Inspectable. You can read the hypothetical passage and see why a result appeared.
• Combines well. It fits beside hybrid search, reranking and other query methods.

Limitations

• Extra latency and cost. Each query needs a generation step, and multi-sample versions need several.
• Dependence on model knowledge. If the language model does not know the topic, the passage is off target. For private jargon, product names and recent events, this is a real risk.
• Topic drift. A confident wrong guess pulls retrieval to the wrong area.
• Poor for exact lookups. Part numbers, IDs, error codes and names are dropped or distorted.
• Variable output. Randomness can make results change between runs unless you fix the settings or cache the passages.
• Bias toward the model's view. The passage reflects what the model thinks an answer looks like, which may differ from what your corpus says.
• Not always better. On well-matched queries and strong in-domain retrievers, HyDE can give little or no gain.

How it compares with alternatives

How to choose

• If queries are short and the corpus is documentation-like, try HyDE early.
• If queries contain many codes and names, prefer hybrid search and keep HyDE on the dense side only.
• If a conversation history makes the question incomplete, rewrite it first, then consider HyDE.
• If you have labeled pairs, a fine-tuned retriever may beat zero-shot tricks.
• If latency budgets are tight, cache passages, use a small model, or use doc2query at index time.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows and columns of mixed numeric, categorical and text cells, with missing values and often no meaningful column names. HyDE is a text retrieval trick, so it does not run inside those models. But the idea behind it, searching with a plausible stand-in for what you want to find, maps onto several tabular tasks, and it can help the systems around such models.

Finding tables, columns and datasets

A user may ask "which table has customer churn by month?" Table descriptions are often short or missing, and the words in the question do not match column names. A HyDE step can generate a hypothetical schema or a description of the table that would answer the question, such as "A monthly table with customer ID, month, churn flag and plan type", and embed that. Matching it against embedded descriptions of real tables and columns finds candidates that plain search misses. This is useful for data catalogs, as covered on pages about data catalog concepts, and for text-to-query systems that need to pick the right tables.

Hypothetical rows for neighbor retrieval

A tabular analog of HyDE is to generate a plausible row, or a plausible filled-in version of a partial row, and use it to look up real rows that resemble it. For in-context learners such as TabPFN, which predict from a labeled table given in the prompt, choosing which real rows to include is a retrieval problem. A synthesized prototype row can serve as a probe to pick neighbors. Note that this is a design pattern you can build around a model. It is not a built-in feature, and the prediction still comes from the model's reasoning over the rows you supply. See /model/architecture for how in-context prediction works.

Imputation

Imputation fills missing cells. A HyDE-like approach proposes a likely value or a likely full row, then searches the real data for rows or reference records that match the proposal, which grounds the imputation in observed data rather than in a guess alone. Methods such as imputation networks and masked cell modeling learn this directly from data, and they are usually the better choice for numeric columns. The retrieval pattern is most useful for text-rich or reference-data cases, such as filling a product category from similar product descriptions.

Anomaly detection

For anomaly detection, the question is often "what would a normal row look like here?" A model can generate a typical profile for a segment, and retrieval can find the real rows nearest to that profile. The distance of an actual row from its nearest normal neighbors then becomes a signal. This is a supporting idea. Dedicated detectors like isolation forest remain the core method for numeric anomalies, and the generated profile must be checked against real data.

Sector and semantic classification

When classifying a table or a column by domain or meaning, as semantic sector heads do, there may be a taxonomy with short labels such as "claims", "inventory" or "telemetry". A short label is a poor match for a long column description. HyDE helps in the other direction: generate a hypothetical description of a column that belongs to each label (or of the label itself), embed it, and compare with the real column description. This gives richer anchors than the bare label and helps zero-shot classification when no labeled examples exist.

Mixed types and missing names

Tabular data has numbers, dates, categories and free text, and some columns have no names. A language model writing hypothetical content needs words to work with, so for such tables it is best to first create a short textual rendering of the table: sampled values, ranges, formats and neighbors. The hypothetical text is then conditioned on that rendering. Mark all generated descriptions as inferred, since the model is guessing meaning from patterns.

Continual learning

Because HyDE changes only the query, it works unchanged as the corpus grows. New tables, new taxonomy entries and new documents enter the index, and the same query-time step finds them without retraining. It complements training-based continual learning such as experience replay: weights hold what the model learned from data, while the index holds knowledge that changes.

Honest limits

HyDE can only help the retrieval around a tabular model. It does not give the model new numeric ability. A generated guess about a table is a hypothesis, and it should never overwrite real values. For measured results on the model side, see /model/research and /model/benchmarks.

Practical guidance and best practices

Start simple

Run plain dense search first. Add HyDE only after you find that short or vague queries fail. Always keep the baseline in your test harness.

Write a corpus-aware prompt

Describe the style of the target documents: documentation, tickets, legal text, scientific abstracts, table descriptions. Ask for a short passage, not a long essay. Around 50 to 150 words works for most cases.

Control randomness

Use a moderate temperature when sampling several passages, so they differ but stay on topic. For a single passage, a low temperature gives stable results. Cache passages by query text to save cost and to make results repeatable.

Anchor with the original query

Average the query vector with the passage vectors, or run a keyword search with the original query beside the dense search. This keeps exact terms in play.

Use a small, fast model where possible

The generation task is easy for most models. A smaller model lowers latency. If the domain is niche, a model with better domain knowledge may matter more than speed.

Route queries

Not all queries need HyDE. A simple router can skip it for queries with IDs, quotes or very specific terms, and use it for short natural-language questions.

Add a reranker

HyDE improves recall, but the top few results may still be noisy. A cross-encoder or late-interaction reranker such as ColBERT can clean up the final order.

Keep generated text out of answers

Do not show the hypothetical passage to the final generator as evidence. Only real retrieved chunks should be cited. Otherwise errors from the guess leak into the answer.

Log everything

Log the query, the passages, the retrieved chunk IDs and the final answer. When a result is bad, the passage tells you whether the guess was the problem.

Evaluate by query type

Break results down into short, long, exact-code and ambiguous queries. Enable HyDE for the groups where it wins. Tools such as RAGAS can help measure retrieval and answer quality, and a judge model can grade answers, but check the judge against a few human labels.

Config sketch

hyde:
  enabled_when:
    min_tokens: 1
    max_tokens: 12
    contains_identifier: false
  generation:
    model: small-fast-model
    temperature: 0.7
    n_samples: 4
    max_words: 120
    style: "internal technical documentation"
  embedding:
    model: same-as-document-index
    include_query_vector: true
  retrieval:
    dense_k: 100
    keyword_k: 100
    fusion: rrf
    rerank_top: 50
    final_k: 10
  cache:
    store: cache.example.internal
    ttl_hours: 24

Prompt patterns by corpus type

The prompt should mimic the form of the documents you hold. A few patterns work well as starting points.

Try two or three prompt styles on a dozen failing queries and keep the one that moves the right chunk up the most. Resist the urge to tune on a single example, since a prompt that fixes one query can break five others.

Latency and cost planning

HyDE adds a generation step in front of retrieval, so plan the budget before you ship it.

• Measure the median and the slow tail of generation time. The tail is what users feel.
• Cap the passage length. Shorter passages are faster and often as useful.
• Generate samples in parallel rather than one after another.
• Cache by normalized query text. Many real queries repeat.
• Skip HyDE through a router when the query is already specific.
• Consider doc2query as an index-time alternative when query latency is the main constraint.

A rough cost model is the number of queries per day times the number of samples times the tokens per passage. Write it down, compare it with the benefit measured on your test set, and revisit it when traffic changes.

Safety and privacy

Queries may contain sensitive text, and they are sent to a generation model. Apply the same data rules you use for any call to a model. Also remember that the generated passage is untrusted text. Never feed it into a place where it could act as an instruction, and never write it into the corpus, since a made-up passage stored as if it were a document would poison later searches.

A short decision checklist

Use this list before turning HyDE on in production.

1. Do failures cluster on short or vague natural-language queries? If not, look elsewhere first.
2. Does the generating model know enough about the domain to write an on-topic guess?
3. Is there a test set with labeled right chunks, split by query type?
4. Is the extra latency inside the budget, including the slow tail?
5. Is the original query kept as an anchor, by vector averaging or a parallel keyword search?
6. Are passages cached and logged?
7. Is there a rule that keeps generated text out of the final answer and out of the corpus?

If the answer to most of these is yes, run an A/B test for a week and compare end-to-end answer quality, not only recall. If the answers are mixed, fix those gaps first.

Typical failure patterns to watch for

Common mistakes

• Using a different embedding model for the passage. The vector must come from the same model as the index.
• Trusting the passage as fact. It is a probe. Never cite it.
• Applying HyDE to every query. Exact lookups can get worse. Route or blend.
• Dropping the original query. Without an anchor, a bad guess sends retrieval far off.
• Long, creative passages. They add noise. Keep them short and on style.
• No caching. Repeated queries pay full cost every time.
• Ignoring domain gaps. A general model may know nothing about internal jargon, so the guess is empty.
• Skipping evaluation. HyDE helps on some corpora and hurts on others; only a test set tells you.
• High temperature for a single sample. One wild passage can derail the search.
• Ignoring privacy. Queries and context are sent to a generation model. Apply the same rules you apply to any model call.

Frequently asked questions

What does HyDE stand for?

HyDE stands for Hypothetical Document Embeddings. The name describes the method: a language model writes a hypothetical document that answers the query, and the system embeds that document and uses its vector to search for real documents. The hypothetical text is only a probe and is not used as an answer.

Why would searching with a made-up answer work better than searching with the question?

A question and its answer are written in different styles and often use different words. An embedding of a made-up answer sits closer to real answer documents than the embedding of a short question does. Even if some details in the guess are wrong, the overall topic and vocabulary usually point to the right region, and the real documents then supply the correct facts.

Does HyDE need training data?

No. It works with an off-the-shelf language model and an off-the-shelf embedding model, which is why it is described as zero-shot. You do not need labeled pairs of queries and documents. You may still want a small test set of your own to check that it helps on your corpus, but that is for evaluation rather than training.

When does HyDE fail?

It fails when the language model knows nothing useful about the topic, so the guess is off target, and when queries rely on exact identifiers such as part numbers or error codes, which the guess may distort or omit. It can also add latency that a tight budget cannot afford. Blending in the original query and routing exact lookups away from HyDE reduces these problems.

How many hypothetical passages should I generate?

Several passages give a more stable result than one, because averaging their vectors smooths out the errors of any single guess. Three to eight is a common range, but the best number depends on cost limits and on how much your queries vary. Test one, three and a larger count on your own data, and keep the smallest number that gives a clear gain.

Is HyDE the same as query rewriting?

They are related but not the same. Query rewriting turns a question into better search text, often a shorter or clearer question or a set of keywords. HyDE writes a full pseudo-answer and uses its embedding. Both are pre-retrieval steps, and they can be combined: rewrite a conversational question into a clear one, then generate the hypothetical passage from the clear version.

Can HyDE be used with keyword search?

The main idea is built for embeddings, because the point is to use the vector of the passage. You can, however, use the passage words as an expanded keyword query, or keep the original query for the keyword side and use the passage only for the dense side. The second option is common in hybrid setups since it protects exact terms.

How does HyDE relate to tabular data?

Tables often have sparse descriptions, so questions like "where is monthly churn stored?" do not match any column name well. A hypothetical table description or row can be embedded and matched against real table and column descriptions, or against real rows. This helps catalogs, schema selection, neighbor lookup and classification support. It supports tabular models from the outside and does not replace their numeric reasoning.

Does the hypothetical document need to be accurate?

No, it only needs to be on topic and written in a style close to the real documents. Accuracy of fine details is not required, because the encoder compresses the passage into a general summary and the real documents provide the facts. Still, a passage that is completely wrong about the topic harms retrieval, so the generating model should have reasonable knowledge of the domain.

How do I decide if HyDE is worth the extra cost?

Compare recall and end-to-end answer quality on a test set with and without it, broken down by query type. If gains on short and vague queries are clear and your latency budget allows one more model call, keep it, perhaps behind a router. If gains are small, put the effort into chunking, hybrid search, contextual chunks or reranking instead.

Glossary

• Averaging (vectors): Combining several embedding vectors into one by taking their mean.
• BM25: A classic keyword ranking function.
• Cache: A store of earlier results reused to save cost.
• Contrastive training: Training an encoder so that matching texts are close and others are far.
• Dense retrieval: Searching by similarity between embedding vectors.
• Doc2query: A method that generates likely questions for each document at index time.
• Embedding: A vector of numbers that represents the meaning of text.
• Encoder: The model that turns text into an embedding.
• Hallucination: Fluent text that is not backed by facts.
• Hypothetical document: The generated, possibly inaccurate passage used as a search probe.
• Nearest-neighbor search: Finding the stored vectors closest to a given vector.
• Pseudo-relevance feedback: Refining a query using the top results of a first search.
• Query expansion: Adding extra text to a query to improve matching.
• Reranker: A model that re-scores retrieved candidates for better ordering.
• Router: A simple component that decides which retrieval path a query takes.
• Temperature: A setting that controls the randomness of generated text.
• Vocabulary gap: A mismatch between query words and document words.
• Zero-shot: Working on a new task or domain without task-specific training examples.

Summary

HyDE searches with an imagined answer instead of the raw question. A language model writes a short hypothetical passage, an embedding model turns it into a vector, and that vector retrieves real documents, which then serve as the evidence. It helps most on short, vague, natural-language queries and in domains without labeled data, and it costs an extra generation step, a dependence on the model's topical knowledge, and weakness on exact identifiers.

For tabular foundation models and table-centric systems, the same pattern supports finding tables and columns from loose questions, probing for neighbor rows, grounding imputation, framing anomaly checks and giving richer anchors for sector and semantic classification. The numeric reasoning stays with the models. Keep the original query as an anchor, never cite the guess, route exact lookups away, and evaluate on your own queries. For the model side, see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to RAG in my stack, under LLMs, retrieval, and evaluation. Others in the same group: