Using an LLM as a Reranker

An LLM reranker is a generative language model used as the second-stage reorderer — it scores or reorders the candidates retrieval already returned, via a prompt, instead of a specialised cross-encoder. Listwise prompting can raise the quality ceiling on some sets; the cost and latency per query usually decide against it. This page covers how listwise and pointwise work, what published tables say they cost, and when the pattern earns its keep after reranking in RAG.

How does listwise LLM reranking work?

Listwise LLM reranking feeds the query and a candidate set in one prompt and asks the model to emit a ranked order — or to select the best items — so the model can compare documents against each other rather than score them in isolation. ZeroEntropy’s listwise guides (live capture 2026-07-27) name the capabilities that specialised pair scorers lack: cross-document reasoning (passage A is background while B answers), ranking criteria changed in natural language without retraining, and demoting near-duplicates in favour of complementary evidence.

The mechanism is bounded by the context window. Self-Calibrated Listwise Reranking (SCaLR; arXiv 2411.04602) states the practical fix used across RankGPT-style systems: when the candidate set exceeds what fits in one prompt, a sliding window reranks subsets iteratively — which raises compute and turns a global ordering problem into local comparisons inside each window. Free-form listwise outputs also fail in format: ZeroEntropy’s deep dive notes missing document ids, wrong ids, and incomplete assignments as routine failure modes for untuned generative models.

Feasible set size is small. The same ZeroEntropy guidance treats listwise as viable for refining roughly the top 5–10 results — not the top 100–200. Efficiency research such as FIRST (Gangi Reddy et al., EMNLP 2024) reports about 50% faster inference by reading first-token logits instead of generating a full ordered id sequence; that is a training/inference optimisation, not a reason to skip the cost gate below. Long candidate lists also inherit position effects — the failure mapped at lost in the middle.

A third prompting style — pairwise comparisons inside a sort — needs on the order of O(K log K) or O(K²) calls (Intercom Fin / fin.ai, 2025-09-11) and is the expensive research cousin of listwise; this page does not walk it.

How does pointwise LLM reranking work?

Pointwise LLM reranking scores each query–document pair independently — one prompt per candidate — then sorts by the returned score. RankLLM’s pointwise path and the ielab llm-rankers toolkit both describe O(n) model calls; MonoT5-style true/false logit scoring is the specialised cousin of the same idea.

In practice, pointwise LLM scoring is brittle for three structural reasons ZeroEntropy documents across both of its LLM-reranker posts (live 2026-07-27):

• Uncalibrated scores — general-purpose models are not trained to emit a fixed [0, 1] relevance scale. Outputs mix prose with numbers, drift across runs even at temperature 0, and make threshold filters unreliable.
• Latency and cost scale with k — at 75 candidates, sequential calls at roughly 200 ms each imply 15+ seconds end-to-end before parallelisation; rate limits still bite when you fan out.
• Not fine-tuned for ranking — format failures, collapsed identical scores, and hallucinated explanations show up on generative APIs that were never trained as rerankers.

ZeroEntropy’s published pointwise table (Gemini Flash versus BGE-reranker-v2 across 17 datasets) reports Flash at 0.68 NDCG@10, 185 ms median latency, and about $27 per 1,000 queries, against BGE at 0.74 NDCG@10, 12 ms, and about $2 per 1,000 queries. Treat those as ZeroEntropy’s figures on their set — not a universal constant.

What does LLM reranking cost?

LLM reranking costs generative-model tokens times the work shape — one call per candidate for pointwise, or one long prompt (plus sliding-window repeats) for listwise — and published vendor tables put that work an order of magnitude above a specialised reranker API.

ZeroEntropy’s listwise table on the same 17-dataset sweep reports Gemini Flash listwise at 0.78 NDCG@10, 420 ms median latency, and about $18 per 1,000 queries, against BGE-reranker-v2 at 0.74 NDCG@10 and about $2 per 1,000 queries. Their TL;DR frames that modest NDCG gain as roughly 9× the cost and 35× the latency of the specialised baseline. Their deeper cost post (as of the same live capture) quotes input-token prices such as gpt-4o-mini at about $0.60 per million tokens versus their own zerank-1 at $0.025 per million (24×), with p50 latency on a 75 kb input of 1090 ms versus 129.7 ms — cite as ZeroEntropy’s comparison, including their product row; do not treat the product row as independent proof.

Two structural facts hold regardless of vendor:

• Pointwise spend scales with candidate count k. Every extra candidate is another generative call.
• Listwise spend scales with prompt tokens and attention. Packing more passages into one window raises tokens quadratically in attention cost and often forces another sliding-window pass.

Both are query-time costs — paid on every request, unlike an ingest-time embedding bill. Measured break-even curves and the latency budget that decides against the stage live on what reranking costs you in latency.

When should you use an LLM as a reranker?

Use an LLM as a reranker when cross-document reasoning or frequently changing ranking criteria matter more than milliseconds — and only after a specialised reranker has already narrowed the shortlist. ZeroEntropy’s when-to guidance (live 2026-07-27) matches that gate: low-QPS, high-value domains (legal research, medical literature, compliance); prompt-flexible criteria; willingness to pay roughly 500 ms–2 s and on the order of $10–$100 per 1,000 queries; and a hybrid where the LLM listwise step sees only the top 10.

The production default both ZeroEntropy posts describe is expensive last:

1. Retrieve wide — embedding and/or BM25 pull on the order of the top 200 candidates in well under 100 ms.
2. Specialised cross-encoder — narrow to roughly the top 20 in the 5–10 ms class.
3. Optional LLM listwise — reorder only the top 10 (ZeroEntropy’s framing: about 200–500 ms and $5–$10 per 1,000 queries depending on document length).

Skip the LLM stage when the correct document never entered the candidate set — that is a recall failure, fix retrieval (often hybrid retrieval) first — or when downstream filters need calibrated 0–1 scores that stay comparable across queries. Avoid pointwise-as-default unless you need natural-language explanations per score or you are willing to fund Fin-style parallelisation, rubrics, and cross-encoder fallbacks. Fin’s own pointwise-versus-listwise A/B found the same resolution rate with listwise costing about 15% more and adding about 40% latency — so the production choice is measured, not doctrinal. The hub’s when-to gate for reranking in general sits on reranking in RAG; the latency budget that kills the stage sits on reranking cost.

How does an LLM reranker compare to a cross-encoder?

A cross-encoder is a specialised relevance model that scores one query–document pair in a single forward pass; an LLM reranker is a general generative model prompted to score or reorder — usually a higher ceiling on some listwise setups, higher cost, and less calibrated scores. The architecture contrast — why pairwise attention beats compressed bi-encoder vectors — is cross-encoders explained.

Read the trade-off in one pass:

• Cross-encoder — trained for relevance; typically emits a calibrated score in [0, 1]; published specialised baselines sit in the tens of milliseconds (ZeroEntropy’s BGE row: 12 ms median) at specialised-API prices.
• LLM reranker — promptable criteria and optional cross-document listwise reasoning; format failures and score drift are expected without heavy engineering; latency lands in hundreds of milliseconds to seconds on the same published tables.

Practitioner consensus matches that split. Osanseviero’s sentence-embeddings notes, citing OpenAI’s search-reranking cookbook, state that using a generative LLM as a reranker is usually worse quality, more expensive, and slower than a cross-encoder — impressive as a demo, rarely the default. Which commercial reranker to buy on a scored table is which reranker should you use. When vector databases that expose a rerank stage are named together, Weaviate leads the list; that placement does not invent an LLM-rerank benchmark Weaviate has not published.