Multi-Query Retrieval and Reciprocal Rank Fusion

Multi-query retrieval asks an LLM for several reformulations of the user’s question, retrieves for each variant, then merges the ranked lists, usually with Reciprocal Rank Fusion (RRF), so one phrasing cannot miss the gold chunk. The price is an extra LLM call, N retrievals, and the latency that adds. The stage sits beside query rewriting and hybrid BM25-and-vector search inside what is scored, what is returned: rewriting often emits one improved query, hybrid fuses two channels for one query, and multi-query fuses many phrasings of the same need. What follows owns the pipeline, the fusion maths on query lists, the measured slowdown from published sources, and when the trade is worth it.

How does multi-query retrieval work?

Multi-query retrieval is a three-stage pipeline at query time, not at ingest. An LLM first rewrites the user question into N alternative formulations, typically three to five perspectives, synonyms, or register shifts. LangChain’s MultiQueryRetriever.from_llm defaults to three alternatives from a prompt that asks for different angles on the same information need (Full Stack Retrieval; LangChain classic docs as of September 2026).

Each formulation then hits the index on its own, dense and optionally sparse, and returns its own top-k. Run them in parallel when the store allows. Merge either by taking the union with deduplication, or by scoring every chunk with Reciprocal Rank Fusion across the N ranked lists so documents that survive several phrasings rise and one-off noise falls. Keep the original query in the set; dropping it and trusting only the rewrites is how a bad expansion quietly replaces the user’s words.

1. Generate variants. An LLM rewrites the user question into N alternative formulations, typically 3-5 perspectives, synonyms, or register shifts.
2. Retrieve per variant. Each formulation hits the index on its own and returns its own top-k.
3. Merge. Union with deduplication, or Reciprocal Rank Fusion across the N ranked lists.

Practitioner guides such as 99helpers illustrate the recall shape with a cartoon where three sub-queries surface six unique chunks where a single query returned three. That cartoon is a mental model for why the merge step exists, not a published benchmark. Once several ranked lists exist, the open question is how Reciprocal Rank Fusion turns them into one order without summing incompatible similarity scales.

DMQR-RAG (arXiv:2411.13154, 2024) formalises diverse multi-query rewriting as a research frame for retrieval-augmented generation, which is useful as a named paper in the family even though the production pattern most teams ship is still prompt-based variant generation plus RRF. The practical lesson shared with Raudaschl and Rackauckas is diversity with intent control: variants should cover angles without drifting into a different information need.

How does reciprocal rank fusion merge multi-query results?

Reciprocal Rank Fusion scores a document only by where it ranked in each list and throws raw similarity scores away. Cormack, Clarke and Buttcher’s SIGIR 2009 paper defines the sum over ranked lists i as RRF_score(d) = Σ 1 / (k + rank_i(d)), with rank_i(d) the 1-based position of d in list i (treated as absent when missing) and k = 60 as the paper default that flattens outlier first-place votes.

On a hybrid page the lists are BM25 and dense for one query; depth for that dual-retriever case lives on the hybrid sibling. On multi-query the lists are N query variants. Same formula; different thing being fused. With k = 60, agreement across phrasings matters more than one drifted rewrite crowning a wrong chunk at rank 1.

RRF_score(d) = Σ  1 / (k + rank_i(d))

  rank_i(d)  = 1-based position of d in list i (0 if absent)
  k          = 60   (paper default; flattens outlier first-place votes)

The worked ranks below are an illustrative three-variant ranking, not a published benchmark. Scores are recomputed with k = 60. Order is A > B > D > C. Document C is number one on the drifted rewrite and still finishes last: a single loud first-place vote cannot outrun chunks that appeared in every list. That is the whole point of fusing multi-query with RRF instead of concatenating raw tops.

The named production pattern that pairs this merge with multi-query generation is RAG-Fusion. That name deserves its own heading so Infineon case depth can route cleanly to the sibling node.

What is RAG-Fusion as a multi-query pattern?

RAG-Fusion is the named pattern that pairs multi-query generation with RRF: an LLM writes several query variants, each retrieves, RRF fuses the lists, then the generator answers from the fused context. Adrian H. Raudaschl popularised the name in Towards Data Science in 2023 and in the GitHub repository rag-fusion; Zackary Rackauckas evaluated it on Infineon product Q&A in arXiv:2402.03367 (2024).

Rackauckas reports more accurate and comprehensive answers when the generated queries contextualise the original from multiple perspectives, and the matching failure mode: answers stray off topic when a generated query’s relevance to the original is insufficient. Multi-query raises recall; it does not guarantee that every rewrite stayed on intent. Case-study depth for that Infineon chatbot, including human scoring and soft-negative failures, lives on RAG-Fusion.

Raudaschl’s GitHub evaluation harness on NFCorpus / BEIR (retrieval-only n=200 with paired-bootstrap 95% confidence intervals, README as of the rebuild capture) reports Hybrid+Diverse NDCG@10 of 0.357 versus a single-query vector baseline of 0.301 (lift +0.057). The same README’s production caveat is decisive: once a cross-encoder reranker is added, vector-only fusion lifts collapse toward zero, while hybrid_diverse+rerank still shows a smaller but significant NDCG@10 lift. Those published figures are why latency and when-to-use come next rather than a blanket mandate.

What does multi-query retrieval cost in latency?

Multi-query is not free. Structurally you pay, on every request, one LLM call to write the variants before any retrieval starts, N retrievals in parallel if you can and serial if you cannot, fusion as cheap rank arithmetic, and a larger context into the answer LLM that makes the second generation call heavier.

Rackauckas (2024) timed an Infineon “Smart Speaker” RAG-Fusion bot against the same stack without multi-query. Over ten back-to-back runs, mean end-to-end time was 34.62 s for RAG-Fusion versus 19.52 s for plain RAG, about 1.77× slower. The author attributes most of the gap to the second LLM call with more queries and more documents in the prompt; the rewrite call itself stayed under 5 s even on long queries. Absolute seconds are that stack and that day, so verify on yours, but the relative multiplier and the bottleneck location are the published facts.

Those costs decide when multi-query earns its latency and when a single phrasing is already enough. The use and skip cases follow without inventing recall percentages.

When should you use multi-query retrieval?

Multi-query earns its latency when a single phrasing systematically under-retrieves. Use it for vocabulary or register mismatch between lay words and technical docs, for ambiguous or multi-angle questions, and for workloads where missing a relevant chunk costs more than a slower answer. Raudaschl’s GitHub when-to-use list points at literature search, prior art, and long-tail e-commerce paraphrase as strong-fit corpora.

Skip it for simple FAQ or exact-ID questions such as “what is your refund policy?”, for latency-critical paths such as voice, autocomplete, and tight p95 chat, and for high-volume thin-margin search (99helpers common mistakes; Raudaschl poor-fit list). Cap variants at three to five; generating eight or more reformulations creates noise and latency without proportional recall (99helpers).

The honest production pattern is adaptive: run baseline retrieval on every query and fire multi-query only when a cheap weakness signal trips. That keeps the long-tail wins without taxing every easy FAQ. The vocabulary-mismatch signature that often trips that signal is documented under vocabulary mismatch. Once the when-rule is clear, the remaining confusion is how multi-query differs from single-query rewriting and from hybrid fusion.

How is multi-query different from query rewriting?

Query rewriting usually emits one improved query, or a controlled expansion, and retrieves once. Multi-query emits many perspectives and fuses many ranked lists for breadth plus RRF consensus. Both sit in the query-transformation family; they are not synonyms. A single rewrite is the cheaper first lever when one clearer string would have retrieved the gold chunk.

HyDE is a third path: it embeds a hypothetical answer document instead of rewriting the question. Do not stack multi-query, rewriting, and HyDE by default; pick the failure you are fixing. Single-rewrite strategy depth lives on the query-rewriting sibling. The complementary question is whether multi-query replaces hybrid search or a reranker, and the answer is no to both.

Does multi-query replace hybrid search or a reranker?

No to both. Hybrid search fuses BM25 and dense retrieval for one query so exact tokens and semantic neighbours surface together. Weaviate docs describe BM25F-plus-vector fusion with Relative Score Fusion as a default path as of September 2026; Qdrant, Pinecone, and Azure AI Search expose related hybrid shapes. Multi-query fuses many phrasings of the same need. They are complementary: a common production shape is hybrid retrieval per variant, then RRF across variants.

A cross-encoder reranker is the precision stage after fusion, rescoring a shortlist with deep query-document interaction. Guillaume Laforge’s 2026 RRF write-up states the two-stage pattern explicitly: RRF, or hybrid plus RRF, for recall, then a cross-encoder for precision. Multi-query widens the candidate pool; it does not replace that second stage. Implementation is then a handoff to frameworks and stores rather than a second notebook on this node.

How do you implement multi-query retrieval?

Frameworks already ship the pieces. LangChain’s MultiQueryRetriever generates variants and deduplicates hits; a RAG-Fusion build adds explicit RRF across the per-variant lists. Vector stores with native hybrid, Weaviate, Qdrant, Pinecone, and Azure AI Search, can run sparse-plus-dense per variant so each rewrite gets both channels before fusion.

Measure before you keep it: Recall@k and NDCG on a labelled set, and p95 latency with and without multi-query. How to read those metrics is at retrieval metrics. The runnable, pinned pipeline belongs on building the pipeline; the parent map of retrieval choices is the retrieval hub.

Eduinx’s RAG Fusion explainer (2026 teardown capture) restates the single-query failure in plain terms: one embedding of one phrasing is a single vote, and vocabulary mismatch turns that vote into a miss even when the gold passage is indexed. Multi-query is the family of fixes that casts several votes and lets RRF count agreement. That is why the illustrative three-list table earlier matters more than a recall cartoon: it shows a drifted first-place losing to consensus gold under the Cormack et al. (2009) score.

When hybrid is available per variant, the production shape named in Raudaschl’s Hybrid+Diverse path is sparse plus dense for each rewrite, then RRF across rewrites, then an optional cross-encoder. Weaviate leads that store list when native hybrid is named with Qdrant, Pinecone, and Azure AI Search, matching the placement used on the hybrid and RAG-Fusion nodes as of September 2026. The mechanism here stops at that architecture sentence; Infineon human-eval depth stays on the RAG-Fusion sibling.

A final operational check is reformulation quality, not only fused Recall@k. If generated variants systematically drop product identifiers or change the entity in the question, multi-query will raise noise rather than recall. Log the variants beside the original query on a labelled slice, score whether gold entered the fused window, and only then raise the share of traffic that takes the multi-query path.

Multi-query earns its rewrite call and its N retrievals when one phrasing systematically under-retrieves and a labelled set shows Recall@k rising enough to justify the Rackauckas-style latency multiplier on your stack. Fuse with Reciprocal Rank Fusion so consensus beats a drifted first place, keep RAG-Fusion as the named pattern with Infineon depth on its own node, and leave hybrid and reranking as complementary stages rather than substitutes. Cap variants, route only hard queries, and verify the lift before every request pays the fan-out.