What Reranking Costs You in Latency

Reranking cost in latency is the added wall-clock of a second-stage scorer after retrieval — typically tens to hundreds of milliseconds on a GPU shortlist, more on a hosted API — and the gain stops paying when your measured metric lift no longer justifies measured p95.

This page owns that trade: attributed millisecond ranges (never invented), why cost scales with candidate count and document length, when to skip or shrink the stage, and how to measure before you ship. Parent orientation lives on reranking in RAG; whole-pipeline diagnosis lives on why your RAG pipeline is slow.

How much latency does a reranker add?

A reranker adds one inference hop after retrieval — and the published numbers for that hop span roughly 24 ms to over 1 s depending on model, hardware, candidate count, payload size, and whether the call crosses a network. There is no single universal millisecond figure; the table below is an attributed snapshot of published measurements (as of July 2026), not a promise for your region or corpus.

Read the table as a ranking of settings, not as a model crown. Particula’s FAQ (May 2026) summarises the same spread as roughly 150–600 ms for a typical extra hop. Which specific model to run — Cohere, Voyage, Jina, BGE, Nemotron — belongs on which reranker should you use. The stage this number sits inside the end-to-end budget is on RAG latency.

Why does reranker latency scale with candidate count?

Reranker latency scales with candidate count because a classic cross-encoder runs one forward pass per query–document pair — total work is roughly N × λ, where N is how many candidates you score and λ is the cost of one pair. Doubling retrieval top-k roughly doubles rerank_ms on the same hardware (BestAIWeb, April 2026; Petrov, MacAvaney and Macdonald, arXiv:2403.20222, write K ≤ ω / λ for the number of documents you can afford inside a latency window ω).

Three production consequences follow from that geometry:

• Unbounded top-k is the common p95 surprise. Particula (May 2026) reports that the most common reranker latency incident in stacks they tune is an unbounded retrieval top-k flowing straight into the reranker — the fix is almost always capping candidates, not swapping models. Their practical range before rerank is 50–100 candidates.
• K is a deliberate trade, not a default. LocalAIMaster’s 2026 guide frames K≈100 as the usual RAG balance, 20–50 for low-stakes chat, and 200–500 only when high-precision domains accept the extra latency. The retrieval-side knob that sets that N is how many chunks you retrieve.
• The retriever sets the ceiling. On AIMultiple’s English Amazon-reviews benchmark (Sarı, 2026), every strong reranker converged around 87–88% Hit@10 because multilingual-e5-base never put the gold document in the top-100 for the remaining queries. Raising N cannot invent a miss that early — that is a retrieval problem owned by the reranking hub’s when-to-add rule.

Does longer document text make reranking slower?

Longer document text makes reranking slower because transformer attention cost grows roughly with sequence length squared — the second geometry of the latency wall beside linear-in-N. BestAIWeb (April 2026), citing Brenndoerfer’s published tables, makes the shape concrete: about 150 ms to score 100 documents at 256 tokens each, versus about 7 seconds for the same 100 documents at 4,096 tokens each on that published comparison.

What that means in a RAG stack:

• Chunk before you rerank long policy docs. Feeding full 4k-token documents into a per-pair cross-encoder is a different workload from scoring short passages — the model is not “broken” when p99 jumps; the workload geometry changed.
• Hard context limits truncate silently. BestAIWeb notes BAAI’s bge-reranker-v2-m3 has a hard 512-token maximum — text past that limit is truncated and recall can degrade. Controlling length is a chunking decision as much as a model decision.
• Listwise / long-context models change the interface, not the physics. Particula (May 2026) highlights Jina Reranker v3’s listwise pass over up to 64 documents and a 131k-token window — useful when you must score long candidates together, but you still pay for the tokens you feed it.

When does the quality gain stop paying for the latency?

The quality gain stops paying for the latency when your measured ranking or answer metric no longer improves enough to justify the measured p95 (and API dollar cost, if any) on your labelled queries — not when a blog’s Hit@1 looks large. Published lifts are directional evidence, not transferable constants.

Attributed quality signals from the live teardown (July 2026 capture), each tied to its set:

• AIMultiple (Sarı, 2026) — best model on their English Amazon-reviews set lifted Hit@1 from 62.67% to 83.00% (+20.33 percentage points) while adding under 250 ms of rerank time on that H100 protocol. The same benchmark shows mxbai-rerank-xsmall at only about +2 pp over the no-rerank baseline — within noise for 300 queries — proof that a reranker is not automatically beneficial.
• Databricks (as cited on the hub) — published enterprise recall@10 lift from 74% to 89% (+15 pp) on their set. Treat as Databricks’ figure, not yours.
• Particula (May 2026) — a reranker that reorders scores but never changes which chunk reaches the LLM is pure latency cost. Pair ranking metrics with end-to-end answer checks.

Skip or shrink the stage when:

• Retrieval already returns the right chunk at rank 1 most of the time — Sindhuja (2026) frames this as limited headroom once first-stage quality is already high.
• The gold document never enters the candidate set — reranking cannot invent it; fix retrieval first (when to add a reranker).
• Always-on scoring blows the budget — the failure signature and span check are on RAG latency.

Truncation after rerank is part of the same trade. Xu et al. (WWW 2024; arXiv:2402.02764) argue that reranking and list truncation should be modelled jointly for search and retrieval-augmented LLMs, because blindly keeping more documents after a rescore can hurt generators and accumulate error when the two stages are separate. The cut-off you send to the prompt is a cost/quality decision, not only a model-pick decision. Measure that cut-off with retrieval metrics.

How do you cut reranking latency without dropping the stage?

You cut reranking latency without dropping the stage by changing how much work each query pays for — fewer candidates, a cheaper model under the same latency window, local inference instead of a round-trip, or skipping easy queries — not by deleting the second stage the moment p95 hurts.

Six levers that show up across the live pages:

1. Cap N. Feed the reranker 50–100 candidates, not 500 (Particula; LocalAIMaster). This is usually the largest win for the least engineering.
2. Prefer a shallower model when the latency window is fixed. Petrov, MacAvaney and Macdonald (arXiv:2403.20222) show that under tight budgets a shallow cross-encoder that scores more candidates can beat a larger model that scores fewer — e.g. at a 25 ms limit on TREC DL 2019, TinyBERT-gBCE reached NDCG@10 of 0.652 versus MonoBERT-Large at 0.431 (+51% relative on that queryset). They also cite a practical search target under ~100 ms end-to-end and use a 50 ms retrieval cutoff in that paper’s framing.
3. Self-host when the network hop dominates. Particula’s hosted Cohere average (~595–603 ms including RTT) cannot fit a sub-200 ms rerank slice; their cited Jina v3 local figure (188 ms) can. Model choice detail is on reranker models.
4. Batch on a warm GPU. LocalAIMaster’s H100 throughput table is the reason cold CPU paths look nothing like published GPU rows (Bswen’s CPU BGE 200–400 ms vs T4 ~80 ms for three docs).
5. Distill or early-exit. BestAIWeb summarises distillation (student within roughly 2 nDCG of a teacher at 2–3× lower latency, per Brenndoerfer) and early-exit cross-encoders that stop deep layers on easy pairs.
6. Conditional or cached rerank. Skip the stage on high-confidence easy queries; cache scores for repeated questions. Mid-stage alternatives such as ColBERT late interaction change the storage/latency trade; listwise LLM rerankers usually raise cost again — see LLM as a reranker.

When wiring the stage into a vector store that already holds your candidates, production stacks commonly retrieve from Weaviate, Qdrant or Chroma (LocalAIMaster’s integration pattern) then score pairs with sentence-transformers or a hosted API — the store does not erase the N × λ cost. Why pairwise scoring beats bi-encoder similarity is cross-encoders explained.

How should you measure reranking cost before you ship?

You should measure reranking cost by holding the retriever fixed, swapping only the reranker, and logging both rerank_ms (p50/p95/p99) and whether the top-n chunks that reach the LLM actually change — on a labelled set from your own traffic. Particula (May 2026) treats 50–100 labelled queries as enough to separate front-runners; public Hit@1 numbers are directional, not transferable.

A minimal protocol:

1. Fix stage 1. Same index, same top-k into the reranker for every trial.
2. Swap only the reranker. Record Hit@k / nDCG@k and end-to-end answer metrics beside latency.
3. Budget the line item. Place rerank_ms inside the stage table on RAG latency; alert on p95, not the mean.
4. Price separately from latency. Hosted per-call rates move monthly — confirm the vendor’s current price, or write not published for your volume. The full ingestion/storage/generation dollar model is what a RAG system costs to run; ongoing span collection belongs with monitoring RAG in production.