Multi-Query Retrieval and Reciprocal Rank Fusion

Multi-query retrieval asks an LLM for several reformulations of the user’s question, retrieves for each variant, then merges the ranked lists — usually with Reciprocal Rank Fusion (RRF) — so one phrasing cannot miss the gold chunk. The price is an extra LLM call, N retrievals, and the latency that adds. This page covers the pipeline, the fusion maths on query lists, the measured slowdown, and when the trade is worth it — next to query rewriting and hybrid search.

How does multi-query retrieval work?

Multi-query retrieval is a three-stage pipeline at query time, not at ingest:

1. Generate variants. An LLM rewrites the user question into N alternative formulations — typically 3–5 perspectives, synonyms, or register shifts. LangChain’s MultiQueryRetriever.from_llm defaults to three alternatives from a prompt that asks for different angles on the same information need (Full Stack Retrieval; LangChain classic docs).
2. Retrieve per variant. Each formulation hits the index on its own (dense, and optionally sparse) and returns its own top-k. Run them in parallel when the store allows.
3. Merge. Either take the union with deduplication, or score every chunk with RRF across the N ranked lists so documents that survive several phrasings rise and one-off noise falls.

Keep the original query in the set. Dropping it and trusting only the rewrites is how a bad expansion quietly replaces the user’s words. 99helpers’ multi-query guide illustrates the recall shape: three sub-queries can surface six unique chunks where a single query returned three — a cartoon, not a benchmark, but the right mental model for why the merge step exists.

How does reciprocal rank fusion merge multi-query results?

Reciprocal Rank Fusion scores a document only by where it ranked in each list — it throws raw similarity scores away. Cormack, Clarke and Büttcher’s SIGIR 2009 paper defines the sum over ranked lists i as:

RRF_score(d) = Σ  1 / (k + rank_i(d))

  rank_i(d)  = 1-based position of d in list i (0 if absent)
  k          = 60   (paper default; flattens outlier first-place votes)

On a hybrid page the lists are BM25 and dense for one query — depth at hybrid search. On multi-query the lists are N query variants. Same formula; different thing being fused. With k = 60, agreement across phrasings matters more than one drifted rewrite crowning a wrong chunk at rank 1.

The table below is an illustrative three-variant ranking (not a published benchmark). Scores are recomputed with k = 60:

Order is A > B > D > C. Document C is #1 on the drifted rewrite and still finishes last: a single loud first-place vote cannot outrun chunks that appeared in every list. That is the whole point of fusing multi-query with RRF instead of concatenating raw tops.

What is RAG-Fusion?

RAG-Fusion is the named pattern that pairs multi-query generation with RRF: an LLM writes several query variants, each retrieves, RRF fuses the lists, then the generator answers from the fused context. Adrian H. Raudaschl popularised the name (Towards Data Science, 2023; GitHub rag-fusion); Zackary Rackauckas evaluated it on Infineon product Q&A in arXiv:2402.03367 (2024).

Rackauckas reports more accurate and comprehensive answers when the generated queries contextualise the original from multiple perspectives — and the matching failure mode: answers stray off topic when a generated query’s relevance to the original is insufficient. Multi-query raises recall; it does not guarantee that every rewrite stayed on intent. Case-study depth for that Infineon chatbot lives on RAG-Fusion.

What does multi-query retrieval cost in latency?

Multi-query is not free. Structurally you pay, on every request:

• One LLM call to write the variants before any retrieval starts.
• N retrievals (parallel if you can; serial if you cannot) instead of one.
• Fusion (cheap — rank arithmetic) and a larger context into the answer LLM, which makes the second generation call heavier.

Rackauckas (2024) timed an Infineon “Smart Speaker” RAG-Fusion bot against the same stack without multi-query. Over ten back-to-back runs, mean end-to-end time was 34.62 s for RAG-Fusion versus 19.52 s for plain RAG — about 1.77× slower. The author attributes most of the gap to the second LLM call (more queries and more documents in the prompt); the rewrite call itself stayed under 5 s even on long queries. Absolute seconds are that stack and that day — verify on yours — but the relative multiplier and the bottleneck location are the published facts.

When should you use multi-query retrieval?

Multi-query earns its latency when a single phrasing systematically under-retrieves:

• Use it for vocabulary or register mismatch (lay words vs technical docs), ambiguous or multi-angle questions, and workloads where missing a relevant chunk costs more than a slower answer (Raudaschl GitHub when-to-use list: literature search, prior art, long-tail e-commerce paraphrase).
• Skip it for simple FAQ or exact-ID questions (“what is your refund policy?”), latency-critical paths (voice, autocomplete, tight p95 chat), and high-volume thin-margin search (99helpers common mistakes; Raudaschl poor-fit list).
• Cap variants at 3–5. Generating eight or more reformulations creates noise and latency without proportional recall (99helpers).

The honest production pattern is adaptive: run baseline retrieval on every query; fire multi-query only when a cheap weakness signal trips. That keeps the long-tail wins without taxing every easy FAQ. The vocabulary-mismatch signature that often trips that signal is documented under vocabulary mismatch.

How is multi-query different from query rewriting?

Query rewriting usually emits one improved query (or a controlled expansion) and retrieves once. Multi-query emits many perspectives and fuses many ranked lists — breadth plus RRF consensus. Both sit in the query-transformation family; they are not synonyms.

HyDE is a third path: it embeds a hypothetical answer document instead of rewriting the question. Do not stack multi-query, rewriting, and HyDE by default — pick the failure you are fixing. Single-rewrite strategy depth lives on query rewriting.

Does multi-query replace hybrid search or a reranker?

No to both. Hybrid search fuses BM25 and dense retrieval for one query so exact tokens and semantic neighbours surface together. Multi-query fuses many phrasings of the same need. They are complementary: a common production shape is hybrid retrieval per variant, then RRF across variants.

A cross-encoder reranker is the precision stage after fusion — rescoring a shortlist with deep query–document interaction. Guillaume Laforge’s 2026 RRF write-up states the two-stage pattern explicitly: RRF (or hybrid+RRF) for recall, cross-encoder for precision. Multi-query widens the candidate pool; it does not replace that second stage.

How do you implement multi-query retrieval?

Frameworks already ship the pieces. LangChain’s MultiQueryRetriever generates variants and deduplicates hits; a RAG-Fusion build adds explicit RRF across the per-variant lists. Vector stores with native hybrid — Weaviate, Qdrant, Pinecone, Azure AI Search — can run sparse+dense per variant so each rewrite gets both channels before fusion.

Measure before you keep it: Recall@k / NDCG on a labelled set, and p95 latency with and without multi-query. The runnable, pinned pipeline belongs on building the pipeline; the parent map of retrieval choices is retrieval in RAG.