Query rewriting

Overview

Query rewriting is the step that reshapes a user's question into one or more better search queries before retrieval runs. Users type short, vague, context-dependent text: "what about last year?", "does it support that?", "fix the error from before". A search index cannot use such text. A rewriting step, usually done by a language model, resolves references, adds missing detail, splits a complex question into parts or generates alternative phrasings. The retriever then works with clean, self-contained queries and finds better documents.

Query rewriting is one of the cheapest and most effective upgrades to a retrieval-augmented generation (RAG) system, because many failures come from the question rather than the index. It pairs with hybrid search, reranking and methods such as HyDE and Contextual Retrieval. The trade-offs are an extra model call per query, the risk of changing the user's intent, and the need to test the rewrites. For table-centric systems, rewriting also turns loose business questions into precise requests about tables, columns, filters and time ranges.

What it is

Query rewriting is any transformation applied to a query before it is sent to a retriever. It sits in the pre-retrieval stage of a RAG pipeline. The input is the user's text, often with the chat history. The output is one or more search-ready queries.

There are several families of rewriting.

• Clarification and de-referencing. Replace pronouns and vague references with the full subject, using the conversation history.
• Normalization. Fix spelling, expand abbreviations and remove filler words.
• Expansion. Add synonyms, related terms or context to widen recall.
• Decomposition. Split a complex question into smaller sub-questions.
• Abstraction (step-back). Ask a more general question to retrieve background knowledge.
• Multi-query. Create several phrasings and merge their results.
• Translation. Convert the query into the language of the documents.
• Structured conversion. Turn the question into filters, a query language or a plan.

An everyday analogy

Imagine you walk into a library and say to the desk clerk, "That one from last time." The clerk cannot help. A good clerk asks questions, remembers that last time you borrowed a book on bread baking, and turns your request into "Show the book on sourdough bread from the cooking section". Query rewriting is the clerk's skill, done automatically.

What it is not

• It is not the retrieval itself. It prepares the input.
• It is not answer generation, though it often uses the same kind of model.
• It is not guaranteed to help. A bad rewrite can lose the user's intent.

Where it came from

The roots of query rewriting go back to classic information retrieval, long before language models. Search engines used spelling correction, stemming, stop-word removal and thesauri to reshape queries. Query expansion with related terms, and pseudo-relevance feedback that uses top results to refine a search, were standard research topics for decades.

With conversational assistants, a new problem appeared: follow-up questions depend on earlier turns. Research on conversational search introduced query reformulation, where a model rewrites the latest turn into a standalone question using the history.

Language models then made flexible rewriting practical. Several named patterns became common in RAG practice.

• Multi-query retrieval, where a model writes several versions of the question.
• RAG-Fusion style pipelines, which combine multi-query with rank fusion.
• Step-back prompting, introduced in 2023 research, which asks the model to form a more general question first.
• Decomposition and least-to-most style prompting, where a hard question is broken into sub-questions.
• Rewrite-Retrieve-Read, a 2023 line of work that treats the query rewriter as a trainable component of the pipeline.
• HyDE, which generates a hypothetical answer to embed (2022).

Frameworks such as LlamaIndex offer query transformation modules, and the idea appears prominently in write-ups on advanced RAG and modular RAG, where rewriting is one of the pre-retrieval modules. In more dynamic systems, an agent decides when and how to rewrite, as described on the agentic RAG page.

How it works

Step 1: Gather the inputs

Collect the latest user message, a few turns of chat history, and any known context such as the user's role, the selected dataset or the current date. Rewriting without context is guesswork.

Step 2: Classify the query

Decide what kind of problem the query has. A small classifier or the model itself can label it.

Step 3: Rewrite with a prompt

A simple standalone rewrite prompt looks like this.

You rewrite the last user message into a standalone search query.
Use the conversation to replace pronouns and vague references.
Keep names, numbers and codes exactly as written.
Do not answer the question. Output the query only.

Conversation:
{{HISTORY}}

Last message: {{MESSAGE}}

Standalone query:

Step 4: Generate variants, if needed

For multi-query, ask for three to five different phrasings that cover different angles, such as synonyms, a more general form and a more specific form.

Step 5: Retrieve with each query

Run the retriever for every rewritten query. Use dense retrieval, keyword retrieval such as BM25, or both.

Step 6: Merge the results

Fuse the ranked lists. Reciprocal rank fusion is a common, simple choice: each document gets a score from its rank in each list, and the scores are summed. Then optionally rerank the merged pool with a cross-encoder or a late-interaction model such as ColBERT.

Step 7: Generate the answer

Give the generator the original question plus the retrieved evidence. Using the original question, not only the rewrite, helps the answer match what the user asked.

The pipeline in one view

user message + history
        |
        v
classify / route
        |
   +----+-----------------+------------------+
   |                      |                  |
   v                      v                  v
standalone rewrite   decomposition      multi-query / step-back
   |                      |                  |
   +----------+-----------+------------------+
              v
     retrieval (dense + keyword)
              v
     fuse -> rerank -> generator (with original question)

Pattern 1: Standalone rewriting for chat

Chat questions are full of references. The rewriter reads the history and produces a question that makes sense alone.

• History: "Tell me about the Atlas export job." then "How often does it run?"
• Rewrite: "How often does the Atlas export job run?"

This is the most common and most valuable pattern. Without it, follow-up questions often retrieve junk.

Pattern 2: Multi-query

Produce several variants and merge.

• Original: "reduce slow dashboards"
• Variants: "improve dashboard load time", "dashboard query performance tuning", "why are dashboards slow".

Each variant has a chance to match differently worded documents. Fusion keeps documents that rank well across variants.

Pattern 3: Decomposition

Split a compound question into steps that can be answered one by one.

• Original: "Which region had the highest churn last year, and how did its support tickets compare to the average?"
• Sub-queries: "churn by region last year", "support ticket volume by region last year", "average support ticket volume per region".

Sub-answers can be retrieved in parallel or in sequence, when a later query depends on an earlier result.

Pattern 4: Step-back

Ask a more general question to fetch principles, then combine with the specific question.

• Original: "Why does job 4471 fail with a lock timeout on node 9?"
• Step-back: "What causes lock timeouts in scheduled batch jobs?"

The general result gives background. The specific query finds the log or ticket.

Pattern 5: Expansion with synonyms and acronyms

Add alternate terms so a keyword index can match. "PTO policy" becomes "PTO paid time off vacation leave policy". Keep the original terms in the mix.

Pattern 6: Translation and normalization

If the corpus is in one language and queries come in another, rewrite into the corpus language. Normalize units, dates and names, for example "last quarter" into an explicit date range when the current date is known.

Pattern 7: Query to structure

Convert the question into a structured request such as filters or a query-language statement. This matters for data stores.

Question: orders over 100 from the north region last month
Structured: {"table": "orders", "filters": [
  {"col": "amount", "op": ">", "val": 100},
  {"col": "region", "op": "=", "val": "north"},
  {"col": "order_date", "op": "between", "val": ["2026-09-01", "2026-09-30"]}]}

Learned rewriters

Instead of prompting a general model, you can train a smaller model to rewrite. Training signals can come from human rewrites or from feedback about which rewrites led to successful retrieval. Learned rewriters can be faster and cheaper at scale, but need data and upkeep. Start with prompting, measure, and train only if volume justifies it.

Why it works

Retrievers match text to text. Their quality depends on how much useful signal the query carries. A rewrite adds signal (resolved references, key terms, structure) and removes noise (filler, ambiguity). It is like turning a mumbled request into a clear order at a restaurant: the kitchen was always capable, but now it knows what to cook.

Key concepts and terms

A worked example

A support assistant has a knowledge base of fictional internal runbooks. A user has this chat.

1. User: "Our nightly export job keeps timing out."
2. Assistant: gives an answer about lock contention.
3. User: "ok and what about the weekly one?"

The last message alone is nearly useless. A search for "what about the weekly one" returns noise.

Step 1: standalone rewrite

The rewriter sees the history and writes: "Why does the weekly export job time out, and how is it different from the nightly export job?"

Step 2: multi-query expansion

Three variants are generated from the standalone question.

• "weekly export job timeout causes"
• "differences between weekly and nightly export jobs"
• "export job lock contention weekly schedule"

Step 3: retrieve and fuse

Each variant retrieves its own top results. Fusion produces a combined list in which a runbook titled "Weekly export: schedule and resource limits" ranks first, because it appears in all three lists.

Step 4: answer

The generator receives the original follow-up, the standalone version and the retrieved runbook text, and answers the actual question.

Code: rewrite, multi-query and fuse

The code below is a sketch with placeholders. call_llm stands for your own model client and search for your own retriever. Hosts are fake.

import json
from collections import defaultdict

REWRITE_PROMPT = """Rewrite the last user message into a standalone search query.
Replace pronouns using the conversation. Keep names, numbers and codes exactly.
Do not answer. Output the query only.

Conversation:
{history}

Last message: {message}

Standalone query:"""

VARIANT_PROMPT = """Write {n} different search queries for the question below.
Cover synonyms, a more general form and a more specific form.
Return a JSON list of strings only.

Question: {question}"""

def call_llm(prompt: str) -> str:
    # Replace with a real call, e.g. to https://llm.example.internal/v1/complete
    raise NotImplementedError

def search(query: str, k: int = 50) -> list[str]:
    # Replace with a real retriever; returns ranked chunk IDs
    raise NotImplementedError

def rrf(rank_lists, c=60):
    score = defaultdict(float)
    for lst in rank_lists:
        for r, cid in enumerate(lst, start=1):
            score[cid] += 1.0 / (c + r)
    return sorted(score, key=score.get, reverse=True)

def retrieve(history: str, message: str, n_variants: int = 3, k: int = 10):
    standalone = call_llm(REWRITE_PROMPT.format(history=history, message=message)).strip()
    try:
        variants = json.loads(call_llm(VARIANT_PROMPT.format(n=n_variants, question=standalone)))
    except json.JSONDecodeError:
        variants = []
    queries = [standalone] + [v for v in variants if isinstance(v, str)]
    lists = [search(q) for q in queries]
    return rrf(lists)[:k], queries

Note the safeguards: the standalone query is always kept, and a parse failure in the variants does not break the pipeline.

Code: a guard against intent drift

A cheap check compares the rewrite with the original to catch drastic changes.

def keeps_key_terms(original: str, rewrite: str) -> bool:
    import re
    # Numbers, codes and capitalized words must survive the rewrite.
    keys = set(re.findall(r"[A-Z][\w-]+|\d[\w.-]*", original))
    return all(k in rewrite for k in keys)

def safe_rewrite(history, message):
    rw = call_llm(REWRITE_PROMPT.format(history=history, message=message)).strip()
    return rw if keeps_key_terms(message, rw) else message

Measuring the effect

Collect real conversations, label the right chunk for each final turn, and compare setups on recall at 5, 10 and 20.

Split results by turn number. Follow-up turns usually show the biggest gain from standalone rewriting. First turns may show little change, and some may get slightly worse if the rewriter adds noise.

Strengths and limitations

Strengths

• Fixes the question, not the index. No re-indexing is needed.
• Big gains for chat. Follow-ups become searchable.
• Handles complexity. Decomposition makes multi-part questions tractable.
• Flexible. It works with dense, keyword and hybrid retrieval.
• Transparent. You can read and log every rewrite.
• Composable. It combines with reranking, HyDE and contextual chunks.

Limitations

• Latency and cost. Each rewrite is a model call, and multi-query multiplies retrieval work.
• Intent drift. A rewrite may add assumptions or lose a key term.
• Hallucinated details. A model may invent a product name or a date to make the query "complete".
• Noise from variants. Poor variants bring irrelevant documents into the fused list.
• Unneeded on good queries. A clear, specific question may be hurt by rewriting.
• Hard to evaluate. Gains depend on query mix, so you need real traffic samples.
• Context leakage. The history may contain sensitive data that is now sent to another model or logged.

How it compares with alternatives

How to choose

• For chat assistants, always start with standalone rewriting.
• For vague, short queries, add multi-query or HyDE.
• For compound questions, add decomposition.
• For corpora with exact codes, keep the original query in the mix and use hybrid search.
• If the intent is genuinely unclear, ask the user a short question instead of guessing.

Role in tabular foundation models

Tabular foundation models learn from raw tables: rows and columns of mixed numbers, categories and text, with missing values and often no helpful column names. Query rewriting is not part of those models. It is a language-side technique, but it matters in the systems around them, because the questions people ask about tables are loose, and the work of turning them into precise requests is exactly what rewriting does.

From business question to table request

A user asks, "How are we doing on returns?" A rewriter can produce a standalone, specific request: "Return rate by month for the last twelve months, by product category". Further steps can map that to the right table, columns and filters. The same pattern as the structured example above applies: the rewrite produces a request that code can execute, and the numbers come from the data, not from the language model.

Finding tables and columns

Table descriptions and column names are short and technical. Rewriting expands user words into the vocabulary of the catalog: "customers who left" becomes "churn, cancellation, subscription end". This helps retrieval over a data catalog or a schema index, and works with keyword and vector search together. When column names are missing or anonymous, retrieval must rely on generated descriptions of the column contents, which should be marked as inferred.

Selecting rows for in-context learning

Models in the family of TabPFN predict from a labeled table placed in the context. When the full table is too large, a subset must be picked. A rewriter can translate a request such as "predict the churn risk for enterprise accounts in the north region" into filters that select the relevant rows. The model then reasons over those rows. See /model/architecture for how the in-context mechanism works. The rewrite chooses the data. It does not do the prediction.

Imputation

To fill a missing value, a system may need to find reference tables or similar rows. A rewritten request, such as "typical shipping cost for fragile items between these two regions", gives a precise lookup into reference data. Learned approaches such as imputation networks and masked cell modeling do the numeric filling from patterns in the table. Rewriting supports the surrounding lookup of reference facts and rules.

Anomaly detection

An analyst might ask, "Show me weird orders from last week." A rewriter makes the request concrete: the time window, the table, the detector to call and the threshold. The detection is done by methods such as isolation forest or a model-based detector, as described on the anomaly detection page. Rewriting is the interface layer. It reduces ambiguity about what "weird" and "last week" mean, and the system should show the interpreted request back to the user.

Sector and semantic classification

When classifying a table or column by domain, as semantic sector heads do, supporting lookups into a taxonomy or glossary benefit from rewriting. A column description like "amt in cents, net of tax" can be rewritten into taxonomy terms such as "monetary amount, net, minor currency unit" before searching reference entries. This gives better matches with the formal vocabulary of the taxonomy.

Time, units and filters

Natural language about tables is full of relative time ("last quarter"), units ("in thousands"), and fuzzy groups ("big customers"). A rewriter that knows the current date and the data dictionary can turn these into explicit ranges, units and thresholds. Make each assumption visible, for example "interpreting big customers as the top 10 percent by revenue", so users can correct it.

Continual learning

Queries change over time, and so do tables and vocabularies. Because rewriting acts on the query, it adapts when the glossary or the catalog is updated, with no need to retrain a model. Logs of failed rewrites become training data for a learned rewriter or for prompt fixes. This complements training-based continual learning such as experience replay.

Honest limits

A rewriter is a language model and can misread a business term. It should not be trusted to compute numbers, and it should not silently change the meaning of a question about data. Keep numeric reasoning in the data layer or in the tabular models, show the interpreted request, and log it. For measured results on the model side, see /model/research and /model/benchmarks.

Practical guidance and best practices

Start with standalone rewriting

If your assistant handles multi-turn chat, this single step gives the largest and most reliable gain. Build it first and measure.

Keep the original query in play

Always retrieve with the original or the standalone query in addition to variants. This guards against a bad rewrite and keeps exact terms available.

Protect key terms

Instruct the model to keep names, numbers, codes and quoted phrases unchanged. Add a code check, as in the guard above, and fall back to the original if key terms vanish.

Use a small, fast model

Rewriting is a short, well-defined task. A small model is often enough, and speed matters because it sits on the critical path. Test for quality on your own queries.

Limit the number of variants

Three to five variants are usually enough. More variants bring more noise and more retrieval load. Test whether each additional variant adds recall.

Route by query type

Skip rewriting for exact lookups and for queries that are already specific. Use a simple classifier or rules.

Pass the date and the dictionary

Give the rewriter the current date, the user's locale and a short glossary of company or domain terms, so it resolves "last month" and local jargon correctly.

Show your interpretation when stakes are high

For data questions, display the rewritten request: "Showing return rate by month, last 12 months, by category." Let the user confirm or edit it.

Log inputs and outputs

Store the original, the history window, each rewrite and the retrieval result. When something fails, you can see at which step.

Evaluate on real traffic

Sample real queries, label the right sources, and compare with and without rewriting by turn number and by query type. Use tools like RAGAS for retrieval and answer metrics, and a judge model for answer quality with human spot checks.

Combine with other methods

Use rewriting with hybrid search, a reranker and, where needed, HyDE. Each handles a different weakness.

Config sketch

query_rewriting:
  router:
    skip_if:
      - contains_identifier: true
      - token_count_over: 25
  standalone:
    history_turns: 6
    model: small-fast-model
    keep_terms: [numbers, codes, quoted_text, capitalized_words]
  multi_query:
    enabled: true
    variants: 3
  decomposition:
    enabled_for: [compound_questions]
    max_sub_queries: 4
  context:
    pass_current_date: true
    glossary: glossary.example.internal/terms
  retrieval:
    dense_k: 50
    keyword_k: 50
    fusion: rrf
    rerank_top: 50
    final_k: 10
  safety:
    fall_back_to_original_on_drift: true
    log_rewrites: true

Common mistakes

• Rewriting every query. Specific, exact queries can get worse.
• Dropping the original. A bad rewrite then has nothing to correct it.
• Letting the model answer. The rewriter should output only a query, not an answer or an explanation.
• Too many variants. Noise rises faster than recall.
• Ignoring history length. Too little history loses references, too much adds distraction.
• Inventing details. Models may add dates, names or constraints that the user never gave.
• No drift check. Without a guard, silent meaning changes go unnoticed.
• Passing only the rewrite to the generator. The generator should also see the user's original wording.
• No logs. You cannot debug what you did not record.
• Treating relative time loosely. "Last quarter" without a date context is a bug waiting to happen.
• Skipping evaluation by turn. First-turn queries and follow-ups behave differently.
• Sending sensitive history to external models without review. Apply your data rules to rewriting calls too.

Frequently asked questions

What is query rewriting in simple terms?

Query rewriting means changing the user's question into a clearer or richer search request before the search runs. A model fills in what is missing, such as what "it" refers to, fixes wording, and sometimes splits one question into several smaller ones. The goal is to give the retriever text it can match well against documents.

Why do follow-up questions need rewriting?

Follow-up questions often depend on earlier turns, so on their own they carry almost no search signal. A phrase like "what about the weekly one?" has no subject. A rewriter reads the conversation and produces a standalone question that names the subject, which turns a hopeless search into a normal one. This is usually the first and most valuable rewriting step.

What is the difference between query rewriting and query expansion?

Query expansion adds related words or phrases to widen matching, while rewriting is the broader idea of changing the query in any helpful way. Rewriting can also resolve references, split a question, translate it or turn it into structured filters. Expansion is one kind of rewriting. Many pipelines use both and keep the original terms so that exact matches still work.

What is multi-query retrieval?

Multi-query retrieval generates several versions of the same question, runs each one through the retriever, and merges the results, often with reciprocal rank fusion. Each version can match differently worded documents, so the merged list tends to cover more ground. It costs extra retrieval calls, and too many variants add noise, so a small number such as three to five is common.

How is query rewriting different from HyDE?

Query rewriting produces better questions or search strings, while HyDE produces a made-up answer and searches with its embedding. Both aim to close the gap between how people ask and how documents are written. They can be combined: rewrite a conversational message into a clear question first, then use HyDE on that clear question if the queries are short and vague.

Can query rewriting make results worse?

Yes. A rewrite can drift from the user's intent, drop a key identifier, or add details that were never asked. Variants can bring in irrelevant documents. To limit this, keep the original query in the retrieval mix, instruct the model to preserve names and codes, check that key terms survive, and measure results with and without rewriting on real queries.

Do I need rewriting if I already use hybrid search?

Hybrid search and rewriting solve different problems. Hybrid search combines keyword and vector retrieval to handle both exact and meaning-based matching, but it still receives whatever query it is given. If the query is vague or depends on chat history, hybrid search cannot repair it. Rewriting fixes the input, so the two work well together.

How does query rewriting help with tables and databases?

People ask loose questions about data, such as "how are returns doing?" Rewriting turns them into precise requests with a table, columns, filters, units and a date range, and it expands user words into catalog vocabulary so the right tables and columns are found. The numbers themselves should come from the data or from tabular models, not from the language model.

Should I use a large model for rewriting?

Usually not. Rewriting is a short and well-defined task, and it sits on the critical path of every query, so speed matters. A small, fast model is often enough, provided you test it on your own queries. A larger model may be worth it for complex decomposition or for specialized domains with heavy jargon.

How do I evaluate a rewriting step?

Collect real queries with the known right sources, run retrieval with and without the rewrite, and compare recall at several depths, split by turn number and query type. Then check end-to-end answers with a rubric or a judge model, backed by human review of a sample. Also read a random set of rewrites to look for drift and invented details.

Glossary

• Abstraction query: A more general question used to fetch background knowledge, also called a step-back query.
• BM25: A classic keyword ranking function.
• Coreference: A reference, such as "it", to something mentioned earlier.
• Decomposition: Splitting a complex question into simpler sub-questions.
• Dense retrieval: Retrieval by similarity of embedding vectors.
• Fusion: Merging ranked lists from several queries or retrievers.
• Intent drift: A change in meaning caused by a rewrite.
• Learned rewriter: A model trained to rewrite queries for a retrieval system.
• Multi-query: Creating several versions of a query and merging the results.
• Normalization: Cleaning text, such as fixing spelling and expanding abbreviations.
• Pre-retrieval stage: The part of a pipeline that prepares the query before search.
• Query expansion: Adding related terms to a query.
• Reciprocal rank fusion: A method that merges lists by summing inverse ranks.
• Reranker: A model that re-scores candidate results.
• Router: A component that chooses which path a query takes.
• Standalone query: A query that is understandable without the conversation.
• Structured query: A request expressed as filters or a query language instead of free text.

Summary

Query rewriting reshapes user questions into clear, self-contained search queries before retrieval. Its main patterns are standalone rewriting for chat, multi-query, decomposition, step-back questions, expansion, translation and conversion to structure. It improves recall at low engineering cost, but adds latency, can drift from the user's intent, and must be tested on real traffic with the original query kept as a safety net.

For tabular foundation models and table-centric systems, rewriting is the interface layer: it turns loose business questions into precise requests about tables, columns, filters, units and dates, helps find the right schemas and reference data, selects rows for in-context prediction, supports imputation lookups and anomaly requests, and links column descriptions to taxonomies for sector classification. The numeric reasoning stays with the data and the models. Show the interpreted request, log it, and evaluate it. For the model side, see /model/research, /model/architecture and /model/benchmarks.

Related

It belongs to RAG in my stack, under LLMs, retrieval, and evaluation. Others in the same group: