Cross-Encoders Explained: Why Reranking Beats Retrieval
Bi-encoder versus cross-encoder scoring, and the accuracy-for-latency trade that defines the two-stage pattern.
A cross-encoder scores a query and a candidate in one transformer pass — no separate embeddings — so it reranks a shortlist more accurately than bi-encoder similarity. Too slow for full-corpus search: retrieve first, then cross-encode the top-k (reranking).
How does a cross-encoder differ from a bi-encoder?
A bi-encoder embeds the query and each document independently and compares the resulting vectors (usually cosine or dot product). A cross-encoder concatenates the pair — typically [CLS] query [SEP] document [SEP] — and runs a single forward pass to a relevance score. Sentence Transformers’ Cross-Encoders documentation (live 2026-07-27) is explicit: a cross-encoder does not produce a sentence embedding, and you cannot pass an individual sentence to it for indexing.
The practical split that every ranking guide on the top-ranking results for bi encoder vs cross encoder repeats (Sentence Transformers, Milvus, Towards Data Science, ZeroEntropy — captured 2026-07-27):
- Bi-encoder — encode documents once at index time; at query time encode the query and run approximate nearest-neighbour search. Fast at corpus scale; loses fine query–document interaction.
- Cross-encoder — score a pre-defined set of pairs at query time. Accurate on that shortlist; cannot precompute an index of document vectors.
Why does a cross-encoder score more accurately than a bi-encoder?
A cross-encoder scores more accurately because joint self-attention lets every query token attend to every document token in the same forward pass. A bi-encoder compresses each text into one vector before any comparison, so negation, contradiction, and exact numeric matches are averaged away (Pinecone, Rerankers and Two-Stage Retrieval; Towards Data Science, Apr 2026).
Two failure modes that joint scoring catches and independent embeddings often miss:
- Contradiction and polarity — a query for “cheap hotels in Tokyo” can rank a “$500/night luxury” page high on topical overlap; a cross-encoder can attend from “cheap” to “$500” and demote it (Towards Data Science’s worked example). The same pattern shows up on “risks of aspirin” versus a benefits-only passage (mbrenndoerfer, Jan 2026).
- Ordinal pair ranking on a real model — the Hugging Face card for cross-encoder/ms-marco-MiniLM-L6-v2 (live 2026-07-27) scores the Berlin population passage about 8.607 and a museums passage about −4.32 for the same query. Those are model logits used for ordering, not a universal 0–1 probability — unless the head applies a sigmoid.
Accuracy comes from joint attention, not from a magic score range
Vendor blogs sometimes show 0–1 scores; MS MARCO MiniLM checkpoints commonly emit unbounded logits. Sort candidates by the score your model actually returns — do not assume calibrated probabilities without checking the model card.
Why can’t you use a cross-encoder for first-stage retrieval?
A cross-encoder cannot precompute document vectors, so every query–document pair needs a full transformer inference at query time — that cost makes full-corpus search impractical. Pinecone’s rerankers chapter (live 2026-07-27) states the scale bound: scoring 40 million records with a small BERT reranker on a V100 GPU would take more than 50 hours per query, while encoder-plus-vector search finishes the same corpus in under 100 ms.
Sentence Transformers makes the combinatorial cost concrete on a smaller set: clustering 10,000 sentences with cross-encoders would require about 50 million pair scores and about 65 hours; a bi-encoder embeds each sentence once in about 5 seconds, then clusters in vector space. That is why stage 1 stays a bi-encoder and/or BM25 on retrieval, and why measured milliseconds-per-candidate belong on what reranking costs you in latency rather than here.
How does retrieve-then-rerank use a cross-encoder?
Retrieve-then-rerank uses a cross-encoder as stage 2 only: a fast retriever returns a wide shortlist, then the cross-encoder scores each (query, candidate) pair and keeps the top-n for the prompt. Sentence Transformers’ “Combining Bi- and Cross-Encoders” section, Pinecone’s two-stage pattern, and LangChain’s cross-encoder reranker docs (all live 2026-07-27) describe the same loop.
The three steps, and what each one owns:
- Retrieve for recall — bi-encoder vector search and/or BM25 returns a candidate set sized for coverage, not for the prompt (owned by retrieval; hybrid recall lives at hybrid search).
- Rerank for precision — the cross-encoder scores every pair in that shortlist and reorders. Practical widths on the top-ranking results: LangChain’s guide retrieves top-20 then keeps top-5; MarkAICode’s BGE walkthrough uses top_k=20 → top_n=3.
- Truncate for the LLM — only the reranked top-n enter generation. A document absent from the shortlist cannot be promoted — raise first-stage recall before tuning the reranker (reranking hub).
What is an MS MARCO cross-encoder?
An MS MARCO cross-encoder is a reranker pretrained on Microsoft’s MS MARCO passage-ranking data — about 500,000 Bing-derived training examples over a corpus of more than 8.8 million passages (Sentence Transformers MS MARCO Cross-Encoders docs, live 2026-07-27) — and released under the cross-encoder/ms-marco-* family on Hugging Face. Nogueira and Cho’s 2019 Passage Re-ranking with BERT paper catalyzed this pattern; the MS MARCO-trained MiniLM checkpoints remain the usual open starting point.
The Sentence Transformers published ladder on a V100 (same table on the Hugging Face MiniLM-L6-v2 card, live 2026-07-27) is the number set to cite — not a vendor blog’s unspecified “better”:
| Model | NDCG@10 | MRR@10 | Docs / sec |
|---|---|---|---|
| ms-marco-TinyBERT-L2-v2 | 69.84 | 32.56 | 9000 |
| ms-marco-MiniLM-L6-v2 | 74.30 | 39.01 | 1800 |
| ms-marco-MiniLM-L12-v2 | 74.31 | 39.02 | 960 |
cross-encoder/ms-marco-MiniLM-L6-v2 is the default open start: nearly the same NDCG@10 as L12-v2 (74.30 vs 74.31) at almost double the throughput (1800 vs 960 docs/sec). TinyBERT-L2-v2 trades 74.30 − 69.84 NDCG@10 points for 9000 docs/sec when latency dominates. Scored comparisons against Cohere, BGE, Jina and other hosted rerankers live on which reranker should you use — this page only locks the MS MARCO open baseline.
When should you skip a cross-encoder reranker?
Skip a cross-encoder reranker when the quality gain cannot pay for one forward pass per candidate — or when reranking cannot fix the failure you actually have.
- Latency budget is too tight — MarkAICode (2026) treats sub-100 ms total budgets as a skip; ZeroEntropy’s bi-vs-cross post (Mar 2026) puts the harder line at sub-10 ms total. Treat both as published thresholds for their stacks, then measure yours on reranking cost and latency.
- The correct document never entered the shortlist — reranking reorders; it cannot invent missing recall. Fix first-stage retrieval (including hybrid search) before adding a cross-encoder.
- First-stage top-k is already clean — MarkAICode’s rule of thumb is to skip when bi-encoder MRR@5 already exceeds 0.85 on a small corpus; verify that number on your labelled set rather than copying it.
When you still need finer matching than a bi-encoder but full cross-encoding is too expensive, ColBERT late interaction sits between the two. When the quality ceiling matters more than cost per query, LLM-as-reranker is the other exit. Start from the reranking hub if you only need the when-to gate, not the architecture.
What is a cross-encoder?
A cross-encoder is a transformer that scores a query and a document together in one forward pass — typically as [CLS] query [SEP] document [SEP] — and outputs a relevance score. Unlike a bi-encoder, it does not produce a reusable embedding you can index; it is used to rerank a shortlist after fast retrieval.
What is the difference between a bi-encoder and a cross-encoder?
A bi-encoder embeds the query and each document independently and compares vectors (cosine or dot product), which lets you precompute document vectors for fast search. A cross-encoder concatenates the pair and runs joint self-attention, which is more accurate on that pair but too slow to score a full corpus. Production RAG uses both: bi-encoder (or BM25) first, cross-encoder second.
Why can't you retrieve with a cross-encoder alone?
Because every query–document pair needs a full transformer inference and nothing is precomputed. Pinecone’s published scale bound (live 2026-07-27): scoring 40 million records with a small BERT reranker on a V100 would take more than 50 hours per query, versus under 100 ms for encoder-plus-vector search. Stage 1 stays fast; the cross-encoder only sees a shortlist.
What is cross-encoder/ms-marco-MiniLM-L6-v2?
It is an open cross-encoder pretrained on MS MARCO passage ranking and published by the Sentence Transformers / Hugging Face cross-encoder family. On the Sentence Transformers V100 table (live 2026-07-27) it scores NDCG@10 74.30 on TREC DL 19 and MRR@10 39.01 on MS Marco Dev at about 1800 docs/sec — nearly identical accuracy to MiniLM-L12-v2 at almost double the throughput.
How many candidates should you pass to a cross-encoder reranker?
Enough that the right chunk is usually in the set, few enough that latency stays inside budget. LangChain’s cross-encoder guide uses retrieve top-20 then keep top-5; MarkAICode’s BGE walkthrough uses top_k=20 → top_n=3 and warns that above about 50 candidates latency grows with little recall gain. Measure on your labelled set — the cost curves live on /reranking/cost.
Is Cohere Rerank a cross-encoder?
Cohere Rerank is a hosted reranker that scores query–document pairs for second-stage ordering — the same job a local cross-encoder does. Treat it as a cross-encoder-style reranker in the two-stage pattern; model choice and latency comparisons against open MS MARCO and BGE checkpoints belong on /reranking/models.