Cross-encoder reranking

Overview

Cross-encoder reranking is a second step in a search pipeline. A fast first stage finds a shortlist of candidate documents, for example the top 50 or 100. A cross-encoder model then reads the query and each candidate together, as one input, and produces a relevance score. The shortlist is sorted by that score, and only the best few results go on to the user or to a language model.

The reason to add this step is accuracy. A first-stage retriever compares compressed summaries of the query and the document, so it can miss fine details. A cross-encoder lets every query word interact with every document word inside the model, which usually gives a clearly better ordering. The price is speed, since the model must run once per query and candidate pair. That is why it is used only on a small shortlist. In retrieval-augmented generation (RAG), reranking raises the precision of the context given to the language model, which often improves answers and cuts cost. See also BM25, hybrid search and ColBERT.

What it is

A cross-encoder is a neural network, usually a transformer, that takes a pair of texts as a single input and outputs one number. In reranking, the pair is a query and a passage. The number is a relevance score. A higher score means the passage is a better answer to the query.

The word "cross" refers to cross-attention between the two texts. Because the query and passage are placed in one sequence, the model can attend from any query token to any passage token and the other way around. It can notice that a negation changes the meaning, that a number in the query matches a number in the text, or that the passage answers a different question with similar words.

The contrast with a bi-encoder

A bi-encoder encodes the query and the document separately, each into a vector. Relevance is then a simple comparison such as cosine similarity. Document vectors can be computed ahead of time and stored in a vector index, so search is fast. But the model never sees the query and the document together, so it loses detail.

A cross-encoder cannot precompute anything per document, because the score depends on the pair. That makes it slow for large collections but much more precise.

An everyday analogy

Think of hiring. A recruiter first screens hundreds of resumes with a quick keyword filter. That is the first stage. Then a hiring manager sits down with each of the top ten candidates and reads their resume next to the job description, thinking about how each part fits. That is the cross-encoder. The manager could not do this for hundreds of people, but for ten it is worth the time, and the decision is far better.

Pointwise, pairwise and listwise

• Pointwise rerankers score each query and document pair on its own. This is the most common design and the one usually meant by "cross-encoder reranking".
• Pairwise rerankers compare two documents for the same query and decide which is better.
• Listwise rerankers see a whole list and produce an ordering. Some large language models are prompted to do this, reading many passages at once.

This article focuses on pointwise cross-encoders, and mentions the others where they differ.

Where it came from

Ranking with neural networks has a long history in information retrieval, and the problem of ordering documents for a query is the core of the field. Before transformers, researchers tried many neural scoring models, but classic methods like BM25 stayed hard to beat.

The turning point came after the release of the BERT model in 2018. Researchers showed that feeding a query and a passage into BERT as one sequence, and training a small output head to predict relevance, gave large gains on passage ranking tasks. This became known as the monoBERT style of reranking. Large public datasets of search queries with relevant passages made training possible, and later work scaled the idea with bigger models and better training data.

Another line of work looked at sequence-to-sequence models that generate a word like "true" or "false" for relevance and use the probability as the score. Others built more efficient models, and some compared bi-encoders, cross-encoders and late-interaction models such as ColBERT, showing the trade-off between cost and accuracy.

More recently, general-purpose large language models have been used as rerankers. They can be prompted to rate passages, or fine-tuned for ranking. Many commercial and open source rerank services now exist. Because versions change quickly, check current documentation before choosing one. For wider context on how retrieval research connects to model research, see research.

How it works

Step 1: Retrieve candidates

A first-stage retriever finds a shortlist. This can be BM25, a dense vector search with an index such as HNSW or FAISS, or a mix in hybrid search. The goal at this stage is recall: the right documents should be somewhere in the list, even if not at the top.

Step 2: Build the input pairs

For each candidate, the system creates one input string by joining the query and the passage with special separator tokens. A typical form is:

[CLS] query text [SEP] passage text [SEP]

The exact special tokens depend on the model. Long passages are truncated to fit the model's maximum input length, so chunk size matters.

Step 3: Run the transformer

The pair is passed through the transformer. In every layer, self-attention lets each token look at every other token in the pair. Query tokens attend to passage tokens and vice versa. This is the step that gives a cross-encoder its power, and also its cost, since the work grows with the square of the combined length.

Step 4: Produce a score

A small head on top of the final representation, often on the first token, turns the hidden state into a single number. Some models give a raw score called a logit, and some pass it through a sigmoid to get a value between 0 and 1. Raw scores are comparable within one query but not always across queries.

Step 5: Sort and cut

The candidates are sorted by score. The system keeps the top k, often 3 to 10, as the final results or as the context for a language model. Some pipelines also drop candidates below a score threshold, so that the model does not receive irrelevant text.

How the model is trained

• Data. Training uses queries with relevant passages (positives) and non-relevant passages (negatives).
• Hard negatives. Negatives that look relevant but are not, often found by a first-stage retriever, teach the model fine distinctions.
• Loss. Common losses include a binary cross-entropy on relevant versus not, a pairwise loss that prefers the positive over a negative, and a listwise loss over a group of candidates.
• Distillation. A large, slow ranker can teach a smaller one, which keeps most of the accuracy at a fraction of the cost.
• Domain fine-tuning. Training on examples from your own domain often gives larger gains than switching to a bigger general model.

Why it is slow, and how to make it faster

The cost per query is roughly the number of candidates times the cost of one forward pass. Ways to reduce it:

• Limit the shortlist to the smallest size that keeps recall.
• Truncate passages to a sensible length.
• Use a smaller or distilled model, or a quantized one.
• Batch the pairs on a GPU.
• Cache scores for repeated queries and documents.
• Run an early-exit approach, where obviously weak candidates are dropped by a cheaper model first.

Key concepts and terms

A worked example

A small numeric walk-through

Suppose a user asks: "Can I return a damaged item after 30 days?" A first-stage retriever returns five passages, ordered by a fast score.

The best answer is passage 4, because it covers damaged items beyond 30 days. A fast retriever put it fourth. A cross-encoder reads the question with each passage. It sees that passage 1 talks about 30 days but not damage, and that passage 4 handles the damaged case and a longer period. Illustrative scores might look like this.

The reranked order puts passage 4 first. The numbers above are made up to show the idea, not measured from any model. If the language model receives only the top two passages, it now gets the key one, and the answer is more likely to be correct.

Python example with a sentence-transformers style API

from sentence_transformers import CrossEncoder

# Placeholder model name; pick a reranker suitable for your language and domain
model = CrossEncoder("example-org/example-reranker")

query = "Can I return a damaged item after 30 days?"
candidates = [
    "Items may be returned within 30 days of delivery.",
    "Damaged items must be reported with photos.",
    "Refunds are paid to the original payment method.",
    "Damaged items can be replaced or refunded at any time within one year.",
    "Shipping takes 3 to 5 days.",
]

pairs = [(query, c) for c in candidates]
scores = model.predict(pairs)

ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
for text, score in ranked[:3]:
    print(round(float(score), 3), text)

The model name above is a placeholder. Use a real reranker from a model hub you trust, and check its license and language coverage.

A full two-stage pipeline in outline

def search(query, k_first=100, k_final=5):
    # Stage 1: fast recall using a keyword and a vector retriever
    sparse = bm25_index.search(query, top_k=k_first)
    dense = vector_index.search(embed(query), top_k=k_first)
    merged = reciprocal_rank_fusion([sparse, dense])[:k_first]

    # Stage 2: precise rerank
    pairs = [(query, doc.text[:1500]) for doc in merged]
    scores = reranker.predict(pairs, batch_size=32)
    order = sorted(range(len(merged)), key=lambda i: scores[i], reverse=True)

    # Optional: drop weak matches
    final = [merged[i] for i in order if scores[i] > MIN_SCORE][:k_final]
    return final

How to measure the gain

1. Collect a set of real queries with labelled relevant passages.
2. Run the first stage alone and record recall at 100 and nDCG at 10.
3. Add the reranker and record nDCG at 10 and the hit rate in the top 3.
4. Record the added latency at the 50th and 95th percentile.
5. Decide whether the gain justifies the cost, and tune the shortlist size.

If recall at 100 is low, no reranker can fix it, since the right document never reaches the second stage. In that case improve the first stage first.

Design patterns for reranking pipelines

Reranking appears in several shapes. Knowing the common ones helps you pick the one that fits.

Single-stage rerank on a small collection

When the collection is small, a few hundred passages, you can score every passage with the cross-encoder. There is no first stage and no recall loss. This is a good setup for a help centre with a few hundred articles or an internal policy set. The cost is linear in the collection size, so it stops working when the collection grows.

Two-stage retrieve then rerank

This is the standard pattern. A fast retriever returns the top 50 to 200. The cross-encoder scores them. The top few are used. Almost every production RAG system that uses reranking has this shape.

Three-stage cascade

For very large collections, a cascade can balance cost and quality.

1. A sparse or dense retriever returns the top 1000.
2. A small, fast reranker cuts the list to 100.
3. A larger, slower reranker cuts the list to 10.

Each stage runs a heavier model on fewer items. The total cost stays low while the last stage keeps high precision.

Rerank plus diversity

Sometimes the top results are near copies of each other. After scoring, a diversity step such as maximal marginal relevance removes results that repeat the same content, so that the context covers more ground. This matters when a language model needs several different facts to answer.

Rerank with metadata

Combine the model score with simple signals such as recency, source quality or access rights. A common approach is to filter by rights and date before scoring, and then add a small boost for fresh documents after scoring. Keep the blend simple and test it, since hand-set weights are easy to get wrong.

Choosing a score threshold

A threshold decides when to return nothing. This is useful in RAG, where passing weak passages to a language model invites made-up answers.

• Collect a set of queries, some with a good answer in the collection and some without.
• Score the top result for each query.
• Plot the scores for the two groups and look for the point where they separate.
• Pick a threshold that gives the precision you need, and accept the loss of recall that comes with it.
• Re-check the threshold when you change the model, because scales differ between models.

If the system returns nothing, the application can say that it does not know, ask a clarifying question or fall back to a wider search. That is often better than a confident wrong answer.

Estimating cost and latency

A rough planning formula:

latency ~ (candidates / batch_size) * time_per_batch
cost per query ~ candidates * tokens_per_pair * price_per_token_or_gpu_second

Tokens per pair is the query length plus the passage length. If your passages average 300 tokens and you rerank 100 candidates, each query processes about 30,000 tokens in the reranker. Halving the shortlist halves the work. Shortening passages from 300 to 150 tokens does the same.

Practical steps:

• Measure time per batch on your hardware with real passage lengths.
• Set a latency budget for the rerank step, such as a fixed number of milliseconds.
• Choose the shortlist size that fits the budget at the 95th percentile of load.
• Review the budget with the rest of the pipeline. See cost and latency budgets.

Troubleshooting guide

Interpreting rerank results in practice

Reading a handful of reranked lists by eye is one of the most useful things a team can do. Pick ten queries, print the first-stage order and the reranked order side by side, and ask three questions.

• Did the reranker move the best passage up?
• Did it move a poor passage up, and if so, why does it look relevant at first glance?
• Are there queries where both orders are bad, which points to a gap in the collection or the chunking?

Keep notes of what you find. Patterns such as "the model prefers longer passages" or "the model ignores dates" become test cases for later releases.

Scoring long documents

A model has a maximum input length. For a long document, there are three common choices.

• Chunk first. Index passages, not full documents, and rerank passages. This is the most common choice.
• Sliding windows. Score overlapping windows of the document and use the highest window score, or a combination of the top ones, as the document score.
• Summaries or headings. Score a short summary or the headings. This is cheaper but may miss detail.

Whichever you choose, evaluate on queries where the answer sits deep in a long document, since those are the cases that expose weak chunking.

Strengths and limitations

Strengths

• High precision. Full interaction between query and document tokens catches subtle matches and mismatches.
• Handles negation and details. It can tell "not covered" from "covered", and a matching number from a near miss.
• Works on top of any retriever. It does not need to change your index. You can add it to a keyword, dense or hybrid system.
• Better context for language models. Fewer irrelevant passages in the prompt means fewer distractions and lower token cost.
• Domain adaptable. A modest amount of labelled data can lift quality on a specific domain.
• Cuts hallucination risk. Passing better evidence reduces the chance that the language model fills gaps with guesses.

Limitations

• Latency and cost. One model call per candidate adds to every query. GPUs help, but not for free.
• Cannot rescue missed recall. It only reorders what the first stage found.
• Input length limits. Long passages must be cut, which can hide the answer.
• Score calibration. Raw scores may not be comparable across queries, so a single global threshold can be unreliable.
• Domain and language gaps. A model trained on web passages may do poorly on legal text, code or a low-resource language without fine-tuning.
• Bias from training data. A reranker may favour phrasing styles it saw in training.
• Operational complexity. One more model to host, version, monitor and test.

How it compares with alternatives

Choosing between them

• If you need to search millions of documents, start with BM25 and a dense retriever, then rerank the shortlist.
• If you want better accuracy without a second model call per pair, consider ColBERT, which keeps per-token vectors and compares them with a cheap operation.
• If you only have a few hundred documents, you can skip the first stage and rerank everything, if latency allows.
• If you need a reason for each ranking, a feature-based learning to rank model is easier to inspect.
• If labelled data is scarce, a general-purpose reranker used as-is is a good starting point.

Related building blocks

Rewriting the question before search can help the first stage. See query rewriting and HyDE. Adding short context to each chunk helps both stages. See contextual retrieval. For the full pattern of retrieval then generation, see advanced RAG and modular RAG.

Role in tabular foundation models

A cross-encoder is a text model. A tabular foundation model works on tables. They meet where a pipeline needs to choose which text, rows or table pieces are worth giving to a model, and the reranking idea carries over well.

Background: what a tabular foundation model needs

A tabular foundation model is pre-trained on a wide range of tables, often including synthetic ones, so that it can predict on a new table without task-specific training. It reads row and column structure, handles mixed types, tolerates missing values, and does not rely on meaningful column names. At inference, a small set of labelled rows acts as context, which is in-context learning. Examples include TabPFN, TabICL and TabSTAR. See the architecture page for how a model of this kind is built.

Where reranking helps

• Selecting context rows. The context window is limited, so a large training table cannot be passed in whole. A first stage can pick a few hundred neighbours of the query row by a cheap similarity. A reranker, trained to judge which neighbours are most informative for the target, can then keep the best few dozen. This is the same two-stage idea as in text search, applied to rows. It is an active design choice and its value should be tested against random or nearest-neighbour selection on your data.
• Matching text fields to a taxonomy. In sector or semantic classification, a row must be assigned to one of thousands of categories. A first stage retrieves a shortlist of plausible categories from their descriptions. A cross-encoder reads the row text and each category description together and picks the best fit. This gives sharper decisions than vector similarity alone. See semantic sector heads.
• Entity matching and deduplication. Two records may describe the same thing with different words. A cross-encoder over serialised record pairs is a standard way to decide if they match, after blocking has produced candidate pairs. See deduplication.
• Imputation with retrieval. To fill a missing text-like value, a pipeline can retrieve similar rows and let a model propose a value. Reranking the retrieved rows improves the evidence. This complements learned approaches such as masked cell modeling and imputation networks.
• Anomaly detection with explanations. After a tabular model flags a row as unusual, a retrieval step can pull similar past cases or documentation, and a reranker orders them so that an analyst sees the most relevant ones first. See anomaly detection.
• Data documentation search. Questions such as "which column holds the order date" are answered by retrieving column descriptions and data dictionary entries. A reranker improves the top results, since many columns have similar names.

How serialisation matters

A cross-encoder needs text. Tabular rows must be turned into short strings, for example "age 42, plan premium, region north, last login 5 days ago". Good serialisation keeps column names where they are meaningful, rounds numbers sensibly, and drops irrelevant fields. A tabular foundation model does not need this conversion, because it reads cells directly. This difference is a good reason to keep each tool in its own role: the cross-encoder judges text relevance, and the tabular model handles numeric structure and mixed types.

What a cross-encoder does not do

It does not model numeric distributions, handle missing values on its own, or learn from a few labelled rows in context in the tabular sense. It should not replace the foundation model for prediction. It improves which pieces of text and which candidate rows reach the model.

For measured results on tabular tasks, see benchmarks. For research context, see research.

Practical guidance and best practices

Size the shortlist

• Start with 50 to 100 candidates from the first stage and measure recall at that size.
• If recall at 100 is already near its ceiling, a larger shortlist only adds cost.
• If recall is low, fix the first stage, for example by adding hybrid retrieval.

Pick the model with care

• Prefer a model trained for your language. Multilingual rerankers exist, but a language-specific one may do better.
• Check the maximum input length and whether it fits your chunk size.
• Test a small and a large model. A distilled model often reaches most of the gain at a lower cost.
• Check the license before using a model in a product.

Prepare the passages

• Chunk documents into coherent passages of a few hundred words.
• Include the title or section heading in the text sent to the reranker, so that short chunks keep their context.
• Remove boilerplate such as navigation text and footers.

Handle scores properly

• Use scores to sort within a query. Do not compare raw scores across queries without calibration.
• If you need a threshold, calibrate it on labelled data, for example by picking the score where precision reaches the level you need.
• Log scores so that you can later study failure cases.

Control latency

• Batch pairs, and use a GPU for large shortlists.
• Set a time budget and return the first-stage order if the reranker times out. Track how often this happens.
• Cache reranker results for popular queries.
• Consider quantization to speed up inference. See cost and latency budgets.

Evaluate and monitor

• Build an offline test set with labelled queries and passages.
• Track nDCG, hit rate in the top 3 and latency for each release.
• For RAG, also score the final answers with an evaluation library such as RAGAS or TruLens. Look at context precision and groundedness to see whether better ranking led to better answers.
• Keep a regression set of failure cases. See regression boards.

Fine-tune when it pays

• Gather click logs or human labels for your domain.
• Use hard negatives from your own first stage.
• Hold out a test set that the model never sees.
• Compare against the general model on the same test set, and only ship if the gain is real.

Common mistakes

• Reranking everything. Running the cross-encoder over a huge candidate list makes latency explode. Use a bounded shortlist.
• Skipping the recall check. If the right document is not in the shortlist, reranking cannot find it.
• Truncating blindly. Cutting a passage at the model's limit can remove the sentence that holds the answer. Chunk with the limit in mind.
• Trusting raw scores as probabilities. Many models output uncalibrated numbers.
• Ignoring the domain. A model trained on web questions can fail on contracts, medical notes or product catalogs.
• No fallback. If the reranker service is down, the search should still work with the first-stage order.
• Evaluating on too few queries. Small samples hide regressions and exaggerate gains.
• Mixing languages without testing. A reranker may handle one language well and another poorly.
• Feeding raw table rows without serialisation. A text reranker needs clean, readable strings, not raw delimited dumps.
• Forgetting to reindex or retrain. When documents or the domain change, the reranker's quality may drift. Monitor it.
• Using it where a filter would do. If a rule such as a date range or a category filter answers the need, apply it before ranking.

Mistakes in evaluation

• Leakage between train and test. If the reranker was fine-tuned on queries that also appear in the test set, the measured gain is inflated. Keep a clean split.
• Only reporting an average. A mean nDCG can hide a group of queries that got worse. Look at the distribution and at the worst cases.
• Comparing models at different shortlist sizes. A gain may come from a larger shortlist, not a better model. Hold the shortlist fixed when comparing models.
• Ignoring the cost side. A model that gives a small gain at ten times the latency is rarely a good trade. Report quality and cost together.

Mistakes in operations

• Silent model changes. Swapping the reranker version without re-running the test set can move results in ways nobody notices until users complain.
• No timeouts. A slow reranker call can block the whole request path. Set a timeout and a fallback.
• Unbounded input. Very long queries or passages can blow up memory. Cap lengths before the model call.
• Logging nothing. Without stored scores and candidate lists, you cannot debug a bad answer later.

Frequently asked questions

What is the difference between a cross-encoder and a bi-encoder?

A bi-encoder turns the query and each document into separate vectors and compares them, which is fast because document vectors are stored ahead of time. A cross-encoder reads the query and one document together in a single pass, which is slower but more accurate because every query word can interact with every document word.

Why not use a cross-encoder for the whole search?

Because it must run once per query and document pair, and nothing can be precomputed. For a collection of millions of documents, that would take far too long and cost too much. A fast first stage reduces the collection to a short list, and the cross-encoder then spends its effort only where it matters.

How many documents should I rerank?

Most teams rerank somewhere between 20 and 200 candidates. The right number depends on how good the first stage is and how much latency you can spend. Measure recall at several shortlist sizes. Pick the smallest size where recall stops improving much, then confirm that latency is acceptable.

Does reranking help RAG?

Usually yes. A language model works better when the passages in its prompt are on topic, and reranking raises the share of useful passages in the top few. It can also let you pass fewer passages, which lowers token cost. Always check with an evaluation set, since gains vary by data and by how good the first stage already is.

Can a large language model act as a reranker?

Yes. A language model can be prompted to score passages or to order a list. This can work well, especially for complex questions, but it is slower and costlier than a dedicated cross-encoder, and its outputs can vary between runs. Many teams use a dedicated reranker for the main path and a language model for hard cases.

Do I need to fine-tune the reranker?

Not always. A general reranker is a good starting point and works well in many domains. Fine-tuning helps when your domain has special vocabulary or when relevance has a local meaning, such as support tickets or legal clauses. Fine-tune only if you have trustworthy labels and a held-out test set to prove the gain.

How does reranking relate to ColBERT?

Both aim to get finer matching than a single vector comparison. ColBERT stores a vector per token and compares query and document tokens with a cheap operation, so it can score many documents quickly. A cross-encoder is more expensive per pair but lets all tokens interact deeply. Some systems use ColBERT for retrieval and a cross-encoder for the final cut.

Can cross-encoder reranking be used with tables?

It can be used on the text parts of table workflows, such as matching rows to category descriptions, resolving duplicate records, or finding the right column documentation. The rows must be serialised into readable text first. For numeric prediction and handling of missing values, a tabular foundation model is the better tool, and the two can be combined.

What metrics should I track?

Track ranking quality with nDCG at 10 and the hit rate in the top 3 or top 5, and track recall of the first stage separately. Also track latency at the median and at the 95th percentile, cost per thousand queries, and the share of queries that fall back to the first-stage order. For RAG, add answer-level scores.

What can go wrong in production?

Common problems are timeouts under load, drift when the data changes, passages cut at the wrong place, and a model that does not fit the domain. Guard against these with a fallback path, regular evaluation on a fixed test set, sensible chunking and monitoring of score distributions over time.

Glossary

• Bi-encoder: A model that creates separate vectors for query and document.
• Blocking: A cheap step that proposes candidate record pairs for matching.
• Candidate list: The shortlist produced by the first stage.
• Chunk: A passage of a document stored and scored as one unit.
• Cross-attention: Attention between tokens of two different parts of the input.
• Cross-encoder: A model that reads a query and a document together.
• Distillation: Training a small model to imitate a larger one.
• First-stage retrieval: The fast step that finds candidates.
• Hard negative: A non-relevant example that looks relevant.
• Hit rate: The share of queries with a relevant result in the top k.
• Hybrid search: A mix of keyword and vector retrieval.
• In-context learning: Learning from examples given in the input, with no weight update.
• Listwise ranking: Ordering a whole list in one pass.
• Logit: A raw score before a sigmoid.
• MRR: Mean reciprocal rank of the first relevant result.
• nDCG: Normalized discounted cumulative gain, a ranking metric.
• Pointwise ranking: Scoring each pair on its own.
• Recall at k: The share of relevant items found in the top k.
• Reranker: A model that reorders a candidate list.
• Serialisation: Turning a table row into a text string.
• Truncation: Cutting input to fit a length limit.

Summary

Cross-encoder reranking adds a precise second look to a fast first-stage search. The model reads the query and each candidate together, scores the pair, and reorders a shortlist. This usually lifts the quality of the top results at the cost of extra latency, so it is applied only to a bounded list of candidates.

It works best with a strong first stage such as BM25, dense retrieval or hybrid search, and it pairs well with RAG evaluation tools like RAGAS. Around tabular foundation models, it helps choose context rows, match text to taxonomies, resolve duplicates and surface documentation, while the tabular model itself handles numbers, categories and missing values. To go deeper on how such models are built and measured, see research, architecture and benchmarks.

Related

It belongs to RAG in my stack, under LLMs, retrieval, and evaluation. Others in the same group: