Cross-Encoders Explained: Why Reranking Beats Retrieval

A cross-encoder scores a query and a candidate together in one transformer pass, so it reranks a shortlist more accurately than bi-encoder similarity alone. Weaviate’s blog post “Using Cross-Encoders as reranker in multistage vector search” (captured 17 September 2026) states the split: bi-encoders produce indexable vectors for fast search, while cross-encoders take a data pair and output a similarity value without producing a reusable document embedding. Sentence Transformers’ Retrieve and Re-Rank documentation describes the same production pattern: retrieve first with a bi-encoder, then rescore the shortlist with a cross-encoder. Vector databases such as Weaviate, Qdrant, and Pinecone own the first stage; the cross-encoder owns the second. Depth for model bake-offs and latency curves sits under reranking in RAG.

How does a cross-encoder differ from a bi-encoder?

A bi-encoder embeds the query and each document independently and compares the resulting vectors, usually with cosine similarity or a dot product. A cross-encoder always consumes a pair: Weaviate’s blog says the input is two items, for example a query and a passage, and the output is a value that indicates how similar those two items are. Hugging Face’s Cross-Encoder organisation card (captured 17 September 2026) shows the practical API as scoring a list of (query, document) pairs, not embedding a lone sentence for an index.

The operational split is why both model types appear in the same pipeline. Bi-encoders compute document vectors at import time, which Weaviate’s blog notes makes approximate nearest-neighbour search efficient at corpus scale. Cross-encoders cannot precompute an index of document vectors, because every score needs the query and the document in the same forward pass. Elastic’s semantic-reranking documentation (captured 17 September 2026) repeats the same contrast under search ranking: cross-encoders and bi-encoders both exist, and cross-encoding is the slower, more precise second stage.

Weaviate’s conclusion on that blog is the pipeline rule in one sentence: store and retrieve with bi-encoders, then continue with a cross-encoder that reranks the retrieved candidates. Once the architectures are distinct, the next question is why the slower model wins on the shortlist.

Why does a cross-encoder score more accurately than a bi-encoder?

A cross-encoder scores more accurately when trained on a representative set because joint attention lets the model read the query and the document together, rather than comparing two independently compressed vectors. Weaviate’s blog cites the Sentence Transformers line of work and states that a well-trained cross-encoder achieves higher accuracy than bi-encoders, with the trade-off that you must score every candidate against the query at search time. Towards Data Science’s April 2026 advanced RAG retrieval article names the information-loss problem on the bi-encoder side: each text is compressed into one vector before any comparison, so fine interactions are averaged away.

A concrete ordinal example sits on the Hugging Face Cross-Encoder card for cross-encoder/ms-marco-MiniLM-L-6-v2 (captured 17 September 2026). For the query “How many people live in Berlin?”, the population passage scores about 8.607 and the museums passage scores about −4.32. Those numbers are model logits used for ordering, not a universal 0 to 1 probability unless the head applies a sigmoid. The same card’s rank example puts urban-area and population passages at the top of a longer Berlin list, which is the behaviour you want before generation: topical but non-answering passages fall.

Towards Data Science’s pipeline table on a worked sample (captured 17 September 2026) reports bi-encoder precision at 85 percent on a twenty-document mix, then cross-encoder precision at 100 percent after keeping ten, with an LLM stage also at 100 percent on five. Treat that table as that author’s sample, not a universal constant; the directional claim matches Weaviate’s accuracy statement. When high bi-encoder similarity still returns the wrong chunk, the failure mode is documented at wrong-chunk ranking misses. Accuracy is not free: the same joint pass that catches polarity is also why a cross-encoder cannot replace the first-stage index.

Why can’t you use a cross-encoder for first-stage retrieval?

A cross-encoder cannot precompute document vectors, so every query-document pair needs a full transformer inference at query time, which makes full-corpus search impractical. Weaviate’s blog states that scoring every object in a collection of thousands or millions with a cross-encoder would take “forever” for a real search application. Towards Data Science (2026) makes the same bound quantitative for a one-million-document corpus: one million forward passes per query is a non-starter even with a small MiniLM.

That is why stage 1 stays a bi-encoder and/or BM25 on retrieval in RAG, and why stage 2 only sees a shortlist. The measured millisecond cost of that shortlist, and how it grows with candidate count, belongs on what reranking costs you in latency. Echelon Edge’s 2026 pipeline ranges, as cited on RAG latency, put typical reranking at 80 to 300 milliseconds when the second stage is on the path. The architectural answer is already settled before those budgets: cross-encoders rerank; they do not replace the index.

How does retrieve-then-rerank use a cross-encoder?

Retrieve-then-rerank uses a cross-encoder as stage 2 only: a fast retriever returns a wide shortlist, then the cross-encoder scores each (query, candidate) pair and keeps the top-n for the prompt. Sentence Transformers’ Retrieve and Re-Rank documentation (captured 17 September 2026) names that pipeline explicitly. Weaviate’s blog draws the same multistage figure: bi-encoder retrieval inside the vector database, then cross-encoder reranking of the candidates. LangChain’s cross-encoder reranker integration docs (captured 17 September 2026) implement the same loop as a document transformer over a base retriever.

The three steps, and what each owns, are retrieve for recall, rerank for precision, then generate. Stage 1 is owned by retrieval; raising candidate recall with BM25 plus dense fusion is hybrid search. Stage 2 is the pairwise scoring pass. Unstructured’s September 2026 reranking guide, which ranks for related queries, shows a practical width as fetch about 30 then keep about 10 after rerank; LangChain’s docs commonly retrieve a wider set then keep a handful for the prompt. Rerank cannot invent a document absent from the shortlist, which is why a missing-document failure is a retrieval fix, not a reranker fix. Parent orientation for when to add the stage sits on the reranking hub.

Weaviate’s model-provider documentation (as of September 2026) wires the same pattern inside the database: configure reranker-transformers, reranker-cohere, or reranker-voyageai, run the initial search, then pass a Rerank argument so the configured model reorders hits. The Transformers module image example uses cross-encoder-ms-marco-MiniLM-L-6-v2, which is the open MS MARCO starting checkpoint.

What is an MS MARCO cross-encoder?

An MS MARCO cross-encoder is a model trained on Microsoft’s Bing-derived passage-ranking data and published under the cross-encoder/ms-marco-* family on Hugging Face. Weaviate’s blog points practitioners at pretrained Sentence Transformers Cross-Encoder models on Hugging Face, including models trained on MS MARCO for general natural-language search, and notes that out-of-domain tasks need fine-tuning. Towards Data Science (2026) treats cross-encoder/ms-marco-MiniLM-L-6-v2 as the default start for most reranking tasks and MiniLM-L-12-v2 when maximising ranking quality matters more than speed.

Hugging Face’s organisation listing (captured 17 September 2026) shows the family that teams actually download: ms-marco-MiniLM-L6-v2, ms-marco-MiniLM-L12-v2, and ms-marco-TinyBERT-L2-v2 among others. Towards Data Science’s local speed sample on a small document set reported MiniLM-L-12-v2 at about 14.3 docs per second and MiniLM-L6-v2 at about 9.9 docs per second on that author’s hardware; those throughput numbers are that author’s sample, not a leaderboard. The scored comparison of hosted Cohere, BGE, Jina, and other production rerankers is which reranker should you use, not a second copy of that bake-off here.

Nogueira and Cho’s 2019 work on passage re-ranking with BERT is the historical starting point for neural cross-encoder reranking that later MS MARCO checkpoints continue. For Weaviate collections using the Transformers reranker module, the docs’ example tag is the MiniLM MS MARCO image, which keeps the open path aligned with the same starting checkpoint. Domain data that does not look like Bing passages still needs fine-tuning, as both Weaviate’s blog and Towards Data Science state.

When should you skip a cross-encoder reranker?

Skip a cross-encoder when the latency budget cannot afford one forward pass per candidate, when the correct document never entered the shortlist, or when first-stage top-k is already clean on your labelled metrics. Meilisearch’s RAG reranking guide (captured 17 September 2026 on the hub query) is explicit that reranking is not always necessary. If recall at a wider cutoff and recall at the prompt cutoff are both low, fix retrieval before you pay for pairwise scoring; if the gold chunk is absent, open missing document.

Prefer a different second stage when the trade-off is not cross-encoder shaped. Token-level late interaction with precomputable document representations is ColBERT and late interaction. Listwise scoring with a generative model, when the quality ceiling matters more than cost per query, is using an LLM as a reranker. The millisecond curves that decide against any always-on second stage are reranking cost and latency. Elastic’s docs also note long-document handling as a separate implementation concern for semantic reranking; that is a product-depth detail, not a reason to skip the architecture when the ranking miss is proven.

The parent when-to gate on the reranking hub is the recall@wide versus recall@prompt test. The mechanism rule is narrower: joint scoring wins on a shortlist, and the shortlist must already contain the answer. Start with MiniLM MS MARCO when you need an open default; move to the model survey when you need a hosted API or a scored bake-off.

A cross-encoder is a pairwise scorer, not a replacement for retrieval. Weaviate’s multistage blog, Sentence Transformers’ Retrieve and Re-Rank docs, Hugging Face’s MiniLM examples, and Towards Data Science’s 2026 guide all describe the same order: bi-encoder or hybrid search for recall, cross-encoder for precision, then generation on a small top-n. Use the stage when labelled metrics show the right chunk is present but poorly ranked; skip it when retrieval missed the document or when measured latency exceeds the gain. Models, ColBERT, LLM reranking, and cost each have a dedicated leaf under reranking.