RAG-Fusion: Multi-Query Retrieval with Reciprocal Rank Fusion

RAG-Fusion generates several LLM reformulations of the user’s question, retrieves for each variant, fuses the ranked lists with Reciprocal Rank Fusion, then answers from the fused context. Adrian H. Raudaschl popularised the name in Towards Data Science in 2023 and in the GitHub repository rag-fusion; Zackary Rackauckas evaluated the pattern on Infineon product Q&A in arXiv:2402.03367 in 2024. The price is an extra rewrite call, N retrievals, and a heavier generation prompt, which is why the pattern sits beside measured latency and failure modes rather than as a default on every query. The ranking set also confuses RAG-Fusion with hybrid “Fusion RAG,” which fuses BM25 and dense search for one query instead of fusing several query lists. The broader multi-query family, including a worked multi-list score table, lives on the multi-query family.

How does RAG-Fusion work?

RAG-Fusion is a four-stage pipeline at query time, not at ingest. An LLM first rewrites the user question into N alternative formulations that capture different angles, synonyms, or register shifts, and the original query stays in the set. Raudaschl’s reference build asks for four or more angles; Rackauckas’s MEMS example expands “Tell me about MEMs microphones” into how-it-works, advantages, and recommended-product framings so one phrasing cannot miss the gold chunk.

Each formulation then hits the index on its own and returns its own top-k. The hybrid variant also runs BM25 beside dense search for each rewrite, which is the Hybrid+Diverse shape in Raudaschl’s GitHub evaluation harness. Reciprocal Rank Fusion scores every document by where it ranked across the N lists and produces one reordered shortlist. The answer LLM finally receives the fused documents plus the original and generated queries and writes the response. Vanilla RAG skips the generate-variants and fuse steps: one query, one ranked list, one generation call.

That breadth is the point of the pattern. A single user phrasing can miss the vocabulary in the corpus even when the right passage is indexed, which is the same failure class documented under vocabulary mismatch. RAG-Fusion pays rewrite and retrieval cost so agreement across phrasings can lift the right chunk into the prompt before generation starts.

Once the shortlists exist, the next question is how Reciprocal Rank Fusion turns N ranked lists into one order the generator can trust without summing incompatible similarity scales.

How does reciprocal rank fusion re-rank RAG-Fusion results?

Reciprocal Rank Fusion scores a document only by its positions across ranked lists and throws raw similarity scores away. Cormack, Clarke and Büttcher’s SIGIR 2009 paper defines the sum over lists i as RRF_score(d) = Σ 1 / (k + rank_i(d)), with rank_i(d) the 1-based position of d in list i (treated as absent when missing) and k = 60 as the paper default that flattens outlier first-place votes. On RAG-Fusion the lists are N query variants, so agreement across phrasings matters more than one drifted rewrite crowning a wrong chunk at rank 1.

Guillaume Laforge’s February 2026 write-up on reciprocal rank fusion in hybrid search makes the same rank-only point for fusing sparse and dense lists, and MongoDB’s reciprocal rank fusion primer shows why raw score sums fail when the scales of BM25 and cosine similarity do not match. Those dual-retriever recipes are the hybrid problem, not the multi-query problem. BM25-plus-dense dual-list RRF belongs on combining BM25 with vectors; a worked three-variant score table lives on multi-query score tables.

The practical consequence for RAG-Fusion is that a rewrite which ranks an off-topic chunk first cannot dominate the fused shortlist unless other variants also promote it. That property is why Raudaschl’s evaluation keeps Hybrid+Diverse as the deployable variant once a cross-encoder enters the stack: rank consensus across channels and phrasings survives a precision stage better than a single strong list.

RRF_score(d) = Σ  1 / (k + rank_i(d))

  rank_i(d)  = 1-based position of d in list i (0 if absent)
  k          = 60   (paper default; flattens outlier first-place votes)

With the score rule fixed, the open question is what Infineon product Q&A actually measured when Rackauckas ran RAG-Fusion end to end against plain RAG.

What did Rackauckas measure with Infineon RAG-Fusion?

Rackauckas (2024) ran an Infineon MEMS-microphone and MOSFET product chatbot and scored answers by hand on accuracy, relevance, and comprehensiveness across three audiences: engineers, account managers, and customers. For “IP rating of mounted IM72D128,” the bot returned IP57 and explained what the digits mean plus the sealed-membrane design, which added context beyond a short forum confirmation. For a sales prompt about selling a 100V OptiMOS Linear FET for power over ethernet, multi-perspective generated queries (market trends, value proposition, pitch) produced an answer that tied IEEE 802.3bt PoE power, ultra-low RDS(on), and customer-needs discovery into one strategy.

Latency is the number most teams need before they copy the pattern. On ten back-to-back “Smart Speaker” runs, mean end-to-end time was 34.62 s for RAG-Fusion versus 19.52 s for plain RAG, about 1.77× slower. The rewrite call stayed under 5 s even on queries of 70 or more words; the author attributes most of the gap to the second LLM call with more queries and more documents in the prompt. Absolute seconds are that stack and that day, so verify the multiplier on your own models and index.

Separately, Raudaschl’s GitHub evaluation harness on NFCorpus / BEIR (retrieval-only n=200 with paired-bootstrap 95% confidence intervals, README as captured for this rebuild) reports Hybrid+Diverse NDCG@10 of 0.357 versus a single-query vector baseline of 0.301 (lift +0.057 [+0.038, +0.077]). The same README’s production caveat is decisive: once a cross-encoder reranker is added, vector-only fusion lifts collapse toward zero, while hybrid_diverse+rerank still shows a smaller but significant NDCG@10 lift of +0.021 [+0.007, +0.036] over baseline plus rerank. How to read NDCG is at retrieval metrics.

What goes wrong with RAG-Fusion?

Rackauckas’s “Challenges of RAG-Fusion” section names four failure modes that show up once the pattern leaves a notebook. Latency rises about 1.77× on the Infineon stack, and the paper’s mitigations are fewer generated queries or a locally hosted LLM to cut API wait. Off-topic answers appear when a generated query drifts from the user’s intent: the answer can be accurate to the rewrite and still wrong for the original question, which is a retrieval win that becomes a product loss.

Prompt sensitivity is sharper than most tip lists admit. A distributor asking whether IM72D128 suits an outdoor camera got nonsense until the prompt said “microphone”; without that word the model treated the part number as a camera itself. Soft negatives are the fourth mode: asked whether a waterproof mic has sleep or wake, the bot said the documents do not mention those features and recommended asking the manufacturer, while the forum expert said flatly that the product has no sleep mode and only a low-power mode. Retrieval-centred bots hedge when the corpus is silent, and that hedge can look like uncertainty rather than a clear negative.

Automatic metrics such as ROUGE and BLEU fit poorly when sales and customer answers can be correct in many shapes. The paper’s method is human scoring on accuracy, relevance, and comprehensiveness, which is slower to run and closer to how a product owner judges the chatbot. Those failure modes decide when the compute is worth paying, which is the next decision.

When should you use RAG-Fusion?

RAG-Fusion earns its compute when a single phrasing systematically under-retrieves. Raudaschl’s GitHub strong-fit list points at terminology mismatch between lay words and technical docs, workloads where missing a relevant document costs more than a slower answer, and exploratory search over specialist corpora such as literature search, prior art, and long-tail e-commerce paraphrase. Cap variants at roughly three to five; generating eight or more reformulations adds noise and latency without proportional recall.

Skip the pattern for FAQ or exact-ID questions, latency-critical paths such as voice, autocomplete, and tight p95 chat, high-volume thin-margin search, and code or identifier lookup, which is Raudaschl’s poor-fit list. Exact tokens are what BM25 keyword search already handles without an extra rewrite call, and hybrid fusion of BM25 with vectors for one query is a different lever than multi-query RRF.

The honest production pattern is adaptive: run baseline retrieval and a reranker on every query, then fire RAG-Fusion only when a cheap weakness signal trips. The vocabulary-mismatch signature that often trips that signal is documented under when user words do not match the documents. Adaptive routing that chooses whether to retrieve at all is a sibling decision on adaptive retrieval, not a substitute for deciding whether to expand the query set.

How is RAG-Fusion different from multi-query and hybrid search?

Three names collide on the ranking set for this query, and they are not synonyms. Multi-query is the family: generate variants, retrieve for each, merge. Depth on that family, including a worked RRF table across three lists, is at multi-query retrieval. RAG-Fusion is the named pattern that pairs that family with explicit Reciprocal Rank Fusion and then generation, which is Raudaschl’s name and Rackauckas’s Infineon evaluation, and the named pattern is what the Infineon evaluation and Raudaschl write-ups refer to as RAG-Fusion.

Hybrid search, sometimes marketed as “Fusion RAG,” fuses BM25 and dense retrieval for one query so exact tokens and semantic neighbours surface together. Guillaume Laforge’s article ranks for RAG-Fusion terms while teaching hybrid RRF first and treating RAG-Fusion as a “Going Further” extension; MongoDB’s reciprocal rank fusion primer is likewise a hybrid and ensemble explainer. Adjacent “Fusion RAG” recipes on related queries are that hybrid shape, not multi-query RRF. Hybrid depth is at hybrid search.

They are complementary rather than competing defaults. A common production shape is hybrid retrieval per variant, then RRF across variants, which is Raudaschl’s Hybrid+Diverse path. HyDE is a third path that embeds a hypothetical answer instead of rewriting the question. Do not stack RAG-Fusion, rewriting, and HyDE by default; pick the failure you are fixing and measure whether the fused shortlist actually moves recall at the depth the generator reads.

How do you implement RAG-Fusion?

The pattern is fixed: prompt an LLM for variants, retrieve N times, fuse with RRF, then generate. LangChain’s multi-query retriever covers variant generation and dedupe; a RAG-Fusion build adds explicit RRF across per-variant lists. LlamaIndex ships reciprocal-rerank fusion examples that AI Engineering Academy walks as a notebook-style method. Guillaume Laforge shows the same RRF idea in LangChain4j for JVM stacks. Vector stores with native hybrid, including Weaviate, Qdrant, Pinecone, and Azure AI Search, can run sparse plus dense per variant so each rewrite gets both channels before fusion.

Weaviate is the default store in worked examples on this site when a collection must answer both keyword and vector queries inside one fused index, using BM25F beside vector search as documented in Weaviate’s hybrid search guides as of September 2026. That does not make Weaviate required for RAG-Fusion; any store that can return ranked lists per variant can feed RRF. What matters is measuring Recall@k and NDCG on a labelled set, and p95 latency with and without the extra rewrite and retrievals, before you keep the pattern in production.

The runnable, pinned end-to-end pipeline belongs on building the pipeline; the precision stage after fusion belongs on reranking; the parent map of retrieval choices is retrieval in RAG. Richard Song’s Epsilla walkthrough shows a node graph for the same five stages, which is a product-shaped illustration of the same mechanism rather than a different algorithm.

RAG-Fusion is multi-query retrieval with explicit Reciprocal Rank Fusion and a generation step on the fused shortlist, popularised by Raudaschl and measured by Rackauckas at roughly 1.77× latency on an Infineon product chatbot. Use it when one phrasing under-retrieves and skip it when exact IDs, tight p95, or a strong hybrid-plus-rerank path already recover the gold chunk. Keep the name clear of hybrid “Fusion RAG,” measure NDCG and latency on your own set, and route family depth to multi-query, dual-retriever depth to hybrid, and the precision stage to reranking.