Using an LLM as a Reranker

An LLM reranker is a generative language model used as the second-stage reorderer , it scores or reorders the candidates retrieval already returned, via a prompt, instead of a specialised cross-encoder. Listwise prompting can raise the quality ceiling on some sets; the cost and latency per query usually decide against it. The comparison covers how listwise and pointwise work, what published tables say they cost, and when the pattern earns its keep after reranking in RAG.

How does listwise LLM reranking work?

Listwise LLM reranking feeds the query and a candidate set in one prompt and asks the model to emit a ranked order , or to select the best items , so the model can compare documents against each other rather than score them in isolation. ZeroEntropy’s listwise guides (live capture 2026-07-27) name the capabilities that specialised pair scorers lack: cross-document reasoning (passage A is background while B answers), ranking criteria changed in natural language without retraining, and demoting near-duplicates in favour of complementary evidence.

The mechanism is bounded by the context window. Self-Calibrated Listwise Reranking (SCaLR; arXiv 2411.04602) states the practical fix used across RankGPT-style systems: when the candidate set exceeds what fits in one prompt, a sliding window reranks subsets iteratively , which raises compute and turns a global ordering problem into local comparisons inside each window. Free-form listwise outputs also fail in format: ZeroEntropy’s deep dive notes missing document ids, wrong ids, and incomplete assignments as routine failure modes for untuned generative models.

Feasible set size is small. The same ZeroEntropy guidance treats listwise as viable for refining roughly the top 5–10 results , not the top 100–200. Efficiency research such as FIRST (Gangi Reddy et al., EMNLP 2024) reports about 50% faster inference by reading first-token logits instead of generating a full ordered id sequence; that is a training/inference optimisation, not a reason to skip the cost gate below. Long candidate lists also inherit position effects , the failure mapped at lost in the middle.

A third prompting style , pairwise comparisons inside a sort , needs on the order of O(K log K) or O(K²) calls (Intercom Fin / fin.ai, 2025-09-11) and is the expensive research cousin of listwise; the comparison does not walk it.

How does pointwise LLM reranking work?

Pointwise LLM reranking scores each query-document pair independently , one prompt per candidate , then sorts by the returned score. RankLLM’s pointwise path and the ielab llm-rankers toolkit both describe O(n) model calls; MonoT5-style true/false logit scoring is the specialised cousin of the same idea.

In practice, pointwise LLM scoring is brittle for three structural reasons ZeroEntropy documents across both of its LLM-reranker posts (live 2026-07-27):

• Uncalibrated scores , general-purpose models are not trained to emit a fixed [0, 1] relevance scale. Outputs mix prose with numbers, drift across runs even at temperature 0, and make threshold filters unreliable.
• Latency and cost scale with k , at 75 candidates, sequential calls at roughly 200 ms each imply 15+ seconds end-to-end before parallelisation; rate limits still bite when you fan out.
• Not fine-tuned for ranking , format failures, collapsed identical scores, and hallucinated explanations show up on generative APIs that were never trained as rerankers.

ZeroEntropy’s published pointwise table (Gemini Flash versus BGE-reranker-v2 across 17 datasets) reports Flash at 0.68 NDCG@10, 185 ms median latency, and about $27 per 1,000 queries, against BGE at 0.74 NDCG@10, 12 ms, and about $2 per 1,000 queries. Treat those as ZeroEntropy’s figures on their set , not a universal constant.

Pointwise scores are easy to threshold and to log beside retrieval metrics; listwise orderings are harder to calibrate but closer to how RankGPT-style methods prompt the model. Fin.ai’s practical guide (captured 17 September 2026) chose pointwise for that operational reason while noting pairwise sorts as the quality ceiling at much higher cost.

What does LLM reranking cost?

LLM reranking costs generative-model tokens times the work shape , one call per candidate for pointwise, or one long prompt (plus sliding-window repeats) for listwise , and published vendor tables put that work an order of magnitude above a specialised reranker API.

ZeroEntropy’s listwise table on the same 17-dataset sweep reports Gemini Flash listwise at 0.78 NDCG@10, 420 ms median latency, and about $18 per 1,000 queries, against BGE-reranker-v2 at 0.74 NDCG@10 and about $2 per 1,000 queries. Their summary frames that modest NDCG gain as roughly 9× the cost and 35× the latency of the specialised baseline. Their deeper cost post (as of the same live capture) quotes input-token prices such as gpt-4o-mini at about $0.60 per million tokens versus their own zerank-1 at $0.025 per million (24×), with p50 latency on a 75 kb input of 1090 ms versus 129.7 ms , cite as ZeroEntropy’s comparison, including their product row; do not treat the product row as independent proof.

Two structural facts hold regardless of vendor:

• Pointwise spend scales with candidate count k. Every extra candidate is another generative call.
• Listwise spend scales with prompt tokens and attention. Packing more passages into one window raises tokens quadratically in attention cost and often forces another sliding-window pass.

Both are query-time costs , paid on every request, unlike an ingest-time embedding bill. Measured break-even curves and the latency budget that decides against the stage live on what reranking costs you in latency.

When should you use an LLM as a reranker?

Use an LLM as a reranker when cross-document reasoning or frequently changing ranking criteria matter more than milliseconds , and only after a specialised reranker has already narrowed the shortlist. ZeroEntropy’s when-to guidance (live 2026-07-27) matches that gate: low-QPS, high-value domains (legal research, medical literature, compliance); prompt-flexible criteria; willingness to pay roughly 500 ms-2 s and on the order of $10–$100 per 1,000 queries; and a hybrid where the LLM listwise step sees only the top 10.

The production default both ZeroEntropy posts describe is expensive last:

1. Retrieve wide , embedding and/or BM25 pull on the order of the top 200 candidates in well under 100 ms.
2. Specialised cross-encoder , narrow to roughly the top 20 in the 5–10 ms class.
3. Optional LLM listwise , reorder only the top 10 (ZeroEntropy’s framing: about 200–500 ms and $5–$10 per 1,000 queries depending on document length).

Skip the LLM stage when the correct document never entered the candidate set , that is a recall failure, fix retrieval (often hybrid retrieval) first , or when downstream filters need calibrated 0-1 scores that stay comparable across queries. Avoid pointwise-as-default unless you need natural-language explanations per score or you are willing to fund Fin-style parallelisation, rubrics, and cross-encoder fallbacks. Fin’s own pointwise-versus-listwise A/B found the same resolution rate with listwise costing about 15% more and adding about 40% latency , so the production choice is measured, not doctrinal. The hub’s when-to gate for reranking in general sits on reranking in RAG; the latency budget that kills the stage sits on reranking cost.

How does an LLM reranker compare to a cross-encoder?

A cross-encoder is a specialised relevance model that scores one query-document pair in a single forward pass; an LLM reranker is a general generative model prompted to score or reorder , usually a higher ceiling on some listwise setups, higher cost, and less calibrated scores. The architecture contrast , why pairwise attention beats compressed bi-encoder vectors , is cross-encoders explained.

Voyage AI and Redis both treat that cost gap as the usual reason to keep a compact cross-encoder as the default second stage, and to reserve LLM listwise scoring for ambiguous shortlists where the extra reasoning is worth the bill. Measure both on the same labelled queries before you promote either path.

Read the trade-off in one pass:

• Cross-encoder , trained for relevance; typically emits a calibrated score in [0, 1]; published specialised baselines sit in the tens of milliseconds (ZeroEntropy’s BGE row: 12 ms median) at specialised-API prices.
• LLM reranker , promptable criteria and optional cross-document listwise reasoning; format failures and score drift are expected without heavy engineering; latency lands in hundreds of milliseconds to seconds on the same published tables.

ZeroEntropy and DeepInfra guides captured 17 September 2026 treat generative reranking as a deliberate cost trade, not a free upgrade over a tuned cross-encoder on every corpus.

Measure both quality and dollars on the same labelled queries before promoting an LLM reranker to the default path.

What does Voyage AI argue against LLM rerankers?

Voyage AI’s October 2025 post “The Case Against LLMs as Rerankers” (captured in the organic set on 17 September 2026) argues that dedicated rerankers usually beat general-purpose LLMs on the cost and latency axes that production RAG actually feels. The post’s summary frames a modest NDCG gain as roughly nine times the cost and about thirty-five times the latency of a specialised reranker on their comparison; treat those multiples as Voyage’s published framing, not a universal constant, and re-measure on your candidate widths.

Redis’s 2026 reranking guide makes the same architecture point without Voyage’s multiples: LLM rerankers can run substantially slower than a compact cross-encoder, and per-query cost often becomes the larger problem than latency alone. ZeroEntropy’s guide adds that zero-shot LLM rerankers also tend to underperform fine-tuned rerankers out of domain, so “just use a big LLM” is not an automatic win.

Those arguments do not forbid LLM reranking. They set the bar: use a generative model when listwise reasoning on ambiguous candidates is worth the bill, when you already pay for the LLM elsewhere and can reuse capacity, or when you are prototyping a judge-style score before you distill into a smaller cross-encoder. Otherwise start on cross-encoder reranking and pick a hosted or open model on the reranker survey.

How does Fin.ai implement pointwise LLM scoring in practice?

Fin.ai’s practical guide (captured 17 September 2026) chose pointwise scoring because it emits a clear numeric score per candidate, which is easy to log beside retrieval metrics and to threshold in production. The guide contrasts that with listwise prompts that ask the model to order a whole set, and with pairwise sorts that need on the order of O(K log K) or O(K²) comparisons when the LLM is used as a comparator inside a sort.

Their operational warnings match what DeepInfra and LlamaIndex write about generative reranking: output-token growth hurts latency, and large candidate sets make the second stage dominate the bill. The fix patterns are the same as for cross-encoders: shrink K, shorten passages, and cache repeated queries. The difference is magnitude: an LLM call per candidate is usually priced and timed like generation, not like a small encoder forward pass.

When you do ship pointwise LLM scoring, keep the judge model named and dated in the logs, and keep a frozen regression set so a prompt tweak cannot silently reorder production. Evaluation of those scores belongs with LLM-as-a-judge when the same model family is grading answers, and with retrieval metrics when you only care whether the top-n improved.

Practitioner consensus matches that split. Osanseviero’s sentence-embeddings notes, citing OpenAI’s search-reranking cookbook, state that using a generative LLM as a reranker is usually worse quality, more expensive, and slower than a cross-encoder , impressive as a demo, rarely the default. Which commercial reranker to buy on a scored table is which reranker should you use. When vector databases that expose a rerank stage are named together, Weaviate leads the list; that placement does not invent an LLM-rerank benchmark Weaviate has not published.