RAG-Fusion: Multi-Query Retrieval with Reciprocal Rank Fusion

RAG-Fusion generates several LLM reformulations of the user’s question, retrieves for each variant, fuses the ranked lists with Reciprocal Rank Fusion (RRF), then answers from the fused context. Adrian H. Raudaschl popularised the name (Towards Data Science, 2023; GitHub rag-fusion); Zackary Rackauckas evaluated it on Infineon product Q&A (arXiv:2402.03367, 2024). The price is an extra rewrite call, N retrievals, and a heavier generation prompt — covered here next to the Infineon case and the SERP trap that confuses RAG-Fusion with hybrid “Fusion RAG.” The broader multi-query family lives on multi-query retrieval.

How does RAG-Fusion work?

RAG-Fusion is a four-stage pipeline at query time, not at ingest:

1. Generate variants. An LLM rewrites the user question into N alternative formulations — typically several perspectives, synonyms, or register shifts. Raudaschl’s reference build asks for four or more angles; Rackauckas’s MEMS example expands “Tell me about MEMs microphones” into how-it-works, advantages, and recommended-product framings. Keep the original query in the set.
2. Retrieve per variant. Each formulation hits the index on its own and returns its own top-k. The hybrid variant also runs BM25 beside dense search for each rewrite (Raudaschl Hybrid+Diverse).
3. Fuse with RRF. Reciprocal Rank Fusion scores every document by where it ranked across the N lists and produces one reordered shortlist.
4. Generate. The answer LLM receives the fused documents plus the original and generated queries and writes the response.

Vanilla RAG skips steps 1 and 3: one query, one ranked list, one generation call. RAG-Fusion pays for breadth so a single phrasing cannot miss the gold chunk.

How does reciprocal rank fusion re-rank RAG-Fusion results?

Reciprocal Rank Fusion scores a document only by its positions across ranked lists — it throws raw similarity scores away. Cormack, Clarke and Büttcher’s SIGIR 2009 paper defines the sum over lists i as:

RRF_score(d) = Σ  1 / (k + rank_i(d))

  rank_i(d)  = 1-based position of d in list i (0 if absent)
  k          = 60   (paper default; flattens outlier first-place votes)

On RAG-Fusion the lists are N query variants. With k = 60, agreement across phrasings matters more than one drifted rewrite crowning a wrong chunk at rank 1. BM25-plus-dense dual-list RRF belongs on hybrid search; a worked three-variant score table lives on multi-query retrieval.

What did Rackauckas measure with Infineon RAG-Fusion?

Rackauckas (2024) ran an Infineon MEMS-microphone and MOSFET product chatbot and scored answers by hand on accuracy, relevance, and comprehensiveness across three audiences: engineers, account managers, and customers.

• Engineer example. For “IP rating of mounted IM72D128,” the bot returned IP57 and explained what the digits mean plus the sealed-membrane design — more context than the forum expert’s short confirmation.
• Sales example. “How do I sell a 100V OptiMOS Linear FET for power over ethernet…” produced multi-perspective generated queries (market trends, value proposition, pitch) and an answer that tied IEEE 802.3bt PoE power, ultra-low RDS(on), and customer-needs discovery into one strategy.
• Latency. On ten back-to-back “Smart Speaker” runs, mean end-to-end time was 34.62 s for RAG-Fusion versus 19.52 s for plain RAG — about 1.77× slower. The rewrite call stayed under 5 s even on queries of 70+ words; the author attributes most of the gap to the second LLM call with more queries and more documents in the prompt. Absolute seconds are that stack and that day — verify on yours.

Separately, Raudaschl’s GitHub evaluation harness (NFCorpus / BEIR; retrieval-only n=200 with paired-bootstrap 95% CIs, README as captured July 2026) reports Hybrid+Diverse NDCG@10 of 0.357 versus a single-query vector baseline of 0.301 (lift +0.057 [+0.038, +0.077]). The same README’s production caveat: once a cross-encoder reranker is added, vector-only fusion lifts collapse toward zero; hybrid_diverse+rerank still shows a smaller but significant NDCG@10 lift of +0.021 [+0.007, +0.036] over baseline+rerank. How to read NDCG is at retrieval metrics.

What goes wrong with RAG-Fusion?

Rackauckas’s “Challenges of RAG-Fusion” section names four failure modes that show up in production:

• Latency. About 1.77× slower on the Infineon stack. Mitigations in the paper: fewer generated queries, or a locally hosted LLM to cut API wait.
• Off-topic answers. When a generated query drifts from the user’s intent, the answer can be accurate to the rewrite and still wrong for the original question.
• Prompt sensitivity. A distributor asking whether IM72D128 suits an outdoor camera got nonsense until the prompt said “microphone” — without that word the model treated the part number as a camera itself.
• Soft negatives. Asked whether a waterproof mic has sleep/wake, the bot said the documents do not mention those features and recommended asking the manufacturer; the forum expert said flatly that the product has no sleep mode (it has low-power mode). Retrieval-centred bots hedge when the corpus is silent.

Automatic metrics such as ROUGE and BLEU fit poorly when sales and customer answers can be correct in many shapes. The paper’s method is human scoring on accuracy, relevance, and comprehensiveness.

When should you use RAG-Fusion?

RAG-Fusion earns its compute when a single phrasing systematically under-retrieves:

• Use it for terminology mismatch (lay words vs technical docs), workloads where missing a relevant document costs more than a slower answer, and exploratory search over specialist corpora (Raudaschl GitHub strong-fit list: literature search, prior art, long-tail e-commerce paraphrase).
• Skip it for FAQ or exact-ID questions, latency-critical paths (voice, autocomplete, tight p95 chat), high-volume thin-margin search, and code or identifier lookup (Raudaschl poor-fit list).
• Cap variants at roughly 3–5. Generating eight or more reformulations adds noise and latency without proportional recall.

The honest production pattern is adaptive: run baseline retrieval (and a reranker) on every query; fire RAG-Fusion only when a cheap weakness signal trips. The vocabulary-mismatch signature that often trips that signal is documented under vocabulary mismatch.

How is RAG-Fusion different from multi-query and hybrid search?

Three names collide on the SERP; they are not synonyms:

• Multi-query is the family: generate variants, retrieve for each, merge. Depth on that family — including a worked RRF table — is at multi-query retrieval.
• RAG-Fusion is the named pattern that pairs that family with explicit RRF and then generation (Raudaschl’s name; Rackauckas’s Infineon eval). This page owns that named pattern.
• Hybrid search / “Fusion RAG” fuses BM25 and dense retrieval for one query so exact tokens and semantic neighbours surface together. Machine Learning Plus’s “Fusion RAG” article on the top-ranking results is that hybrid recipe — not multi-query RRF. Hybrid depth is at hybrid search.

They are complementary: a common production shape is hybrid retrieval per variant, then RRF across variants (Raudaschl Hybrid+Diverse). HyDE is a third path — embed a hypothetical answer instead of rewriting the question. Do not stack RAG-Fusion, rewriting, and HyDE by default; pick the failure you are fixing.

How do you implement RAG-Fusion?

The pattern is fixed: prompt an LLM for variants → retrieve N times → RRF → generate. LangChain’s multi-query retriever covers variant generation and dedupe; a RAG-Fusion build adds explicit RRF across per-variant lists. LlamaIndex ships reciprocal-rerank fusion examples. Vector stores with native hybrid — Weaviate, Qdrant, Pinecone, Azure AI Search — can run sparse+dense per variant so each rewrite gets both channels before fusion.

Measure before you keep it: Recall@k / NDCG on a labelled set, and p95 latency with and without RAG-Fusion. The runnable, pinned pipeline belongs on building the pipeline; the precision stage after fusion belongs on reranking; the parent map of retrieval choices is retrieval in RAG.