Retrieve and rerank with a cross-encoder
Recall first, precision second. A wide, fast retrieval is followed by a slower, sharper judge that reorders the shortlist; the arrows show how far each chunk moves.
Interactive simulation. It needs JavaScript; the text below explains what it shows.
What you are looking at
Step 1 retrieves eight candidates by vector similarity. Step 2 scores each candidate with a cross-encoder-style judge (here: how much of the question the passage covers, plus phrase matches), and step 3 keeps the best three. All RAG simulations share one knowledge base of 18 chunks from 11 documents of a fictional outdoor-gear shop, and three questions: one phrased differently from the documents, one that needs two documents, and one about the whole corpus.
Things to try
- Watch the arrows: which chunks move up, which fall out?
- Compare the final three with naive RAG for the same question.
- Note that reranking can only reorder what retrieval found; it cannot fix a miss.
How it works in real systems
Rerankers such as BGE-reranker, Cohere Rerank or ms-marco cross-encoders typically rescore 20 to 100 candidates and add tens to a few hundred milliseconds. This site's own AI Search uses the same two-stage design.
Code lab
The same idea in real code, step by step. Copy each block into a notebook or a file and run it in order.
Step 1 · Index the knowledge base
Install the libraries and index a small knowledge base with a sentence-embedding model. Normalised embeddings make the dot product equal to cosine similarity.
pip install sentence-transformers rank-bm25 numpy openai
import numpy as np
from sentence_transformers import SentenceTransformer
chunks = [
"Most items can be returned within 30 days of delivery for a full refund.",
"Damaged or defective items qualify for a full refund or a free replacement; send a photo of the damage to support first.",
"Refunds are issued within five business days after the returned item reaches our warehouse.",
"Parcels damaged in transit must be reported to the carrier within 48 hours of delivery.",
"The Ridge 2 tent weighs 1.9 kg and uses Aurora ripstop fabric for its fly sheet.",
"Aurora ripstop fabric is woven by Lumen Textiles at its mill in Porto, Portugal.",
"Customer complaints in Q3 were mostly about late deliveries during the September peak.",
]
emb = SentenceTransformer("all-MiniLM-L6-v2") # 384 dimensions, small and fast
E = emb.encode(chunks, normalize_embeddings=True)
def dense_search(q, k=3):
s = E @ emb.encode([q], normalize_embeddings=True)[0]
top = np.argsort(-s)[:k]
return [(int(i), float(s[i])) for i in top]
Step 2 · Retrieve wide, then rerank with a cross-encoder
The cross-encoder reads question and passage together, so it is much more precise than comparing two separate vectors, and much slower: use it only on the shortlist.
from sentence_transformers import CrossEncoder
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def retrieve_rerank(q, wide=8, k=3):
cand = [i for i, _ in dense_search(q, wide)]
scores = ce.predict([(q, chunks[i]) for i in cand])
order = np.argsort(-scores)
for r, j in enumerate(order):
print(f"#{r+1} (was #{j+1}) {scores[j]:.2f} {chunks[cand[j]][:60]}")
return [cand[j] for j in order[:k]]
retrieve_rerank("Can I get my money back if my tent arrived torn?")
Step 3 · Budget the latency
import time
t = time.time(); retrieve_rerank("Who makes the Ridge 2 fabric?", wide=50); print(round((time.time() - t) * 1000), "ms for 50 candidates")
Rerank 20–100 candidates; beyond that, recall gains are small and latency grows linearly.