Multi-Hop RAG: Answering Questions That Need Several Documents

Multi-hop RAG is an iterative retrieve-and-reason architecture for questions that need evidence spread across multiple documents, fetching evidence in more than one hop and stopping only when the answer is sufficiently grounded or the hop budget is exhausted. The loop is retrieve, reason, then retrieve again with an updated sub-query, not one retrieve-then-generate pass. Tang and Yang’s MultiHop-RAG benchmark (arXiv:2401.15391, 2024) and practitioner guides such as Cobus Greyling’s Multihop-RAG note treat the same composition problem: a single top-k list rarely contains every supporting document the answer needs. Termination is part of the mechanism, not an afterthought. Diagnosis of hop-miss versus composition miss lives on multi-hop failure diagnosis; the family map is at the named patterns.

What is multi-hop RAG?

Multi-hop RAG is a retrieval-augmented generation design where the system issues more than one retrieval call because the question’s supporting facts sit in different documents, and each hop’s reasoning depends on what the previous hop actually returned. Cobus Greyling’s Multihop-RAG write-up frames the pattern through the MultiHop-RAG dataset: questions whose evidence is distributed across news articles rather than concentrated in one passage. Vectorize.io’s multi-hop implementation guide and Ganesh Kannappan’s Beyond Naive RAG Medium article describe the same iterative retrieve-and-compose shape for production stacks.

Single-hop RAG assumes one retrieval set is enough. Multi-hop RAG assumes composition: the model must combine facts A and B that never co-occur in one chunk. That is why MultiHop-RAG (Tang and Yang, 2024) and related multi-hop QA literature treat hop count as a first-class property of the question, not only of the pipeline. Graph traversal methods that walk entity edges instead of issuing textual hops are a sibling pattern on GraphRAG, not a substitute definition here.

The architecture owns the loop and its stop rule. The failure sibling owns how to tell hop-miss from composition miss when answers still go wrong. Keep those jobs separate when you instrument evaluation.

How does multi-hop RAG work?

Multi-hop RAG turns retrieval into a loop: each hop retrieves candidate evidence for the current sub-question, the model composes or refines a partial answer, then the system decides whether another hop is needed until the termination rule fires or the max hop budget is reached. DSPy’s multihop_search tutorial and Awesome-RAG-style multi-hop examples describe the same retrieve-reason-retrieve progression with an explicit hop counter.

The loop stays grounded because the next retrieval query is conditioned on evidence already in hand, not on the original user question alone. That conditioning is why hop two often looks like a different keyword or entity string than hop one. Zyphra’s graph-based multi-hop QA discussion and APXML-class iterative RAG course notes both stress that the second retrieval is informed by the first hop’s intermediate answer, not a blind re-issue of the same query.

Methods such as IRCoT and FLARE vary when and how the next query is written, but they still sit inside an iterative retrieve-and-reason family. Depth on those named methods lives on their own architecture leaves; this node owns the shared loop skeleton and its stop criterion.

How does query decomposition help multi-hop retrieval?

Query decomposition helps multi-hop retrieval by splitting a single user question into a set of smaller sub-questions so each hop can retrieve the specific evidence piece the next reasoning step depends on. Instead of asking the index for the whole answer at once, the system asks for fact A, then uses A to ask for fact B.

Decomposition reduces retrieval ambiguity: each sub-query targets one evidence type or document requirement. It also makes hop boundaries explicit, so you can log which sub-query produced which passages and how the composition step used them. Bounded loops pair naturally with max hops and sufficiency checks once those sub-queries exist. The dedicated mechanism for that first step is query decomposition.

Without decomposition, multi-hop systems often re-retrieve with a lightly rewritten full question and still miss the second supporting document. Decomposition is not optional window dressing; it is how hop boundaries become measurable.

When should multi-hop retrieval stop?

Stop criterion is the difference between a helpful multi-hop loop and an expensive one. In practice you choose among three termination regimes, including the value-based Stop-RAG style objective in Jeong et al. (arXiv:2510.14337) that treats stopping as a decision over whether another hop improves the answer enough to justify the cost.

Fixed max hops caps the loop by budget so worst-case latency stays predictable. Many production sketches default to two or three hops because FAIR-RAG-class analyses (arXiv:2510.22344) and practitioner guides treat that range as the common cost-benefit band, though your corpus may need a different cap. LLM self-stop asks the model to decide sufficiency against retrieved evidence and the partial answer state; use it only when you trust that self-evaluation. Value-based termination uses a learned or scripted stop decision that predicts whether another hop improves the answer enough to pay for itself.

Without an explicit stop rule, multi-hop designs can loop until token budgets or timeouts cut them off, which is latency failure dressed as thoroughness. Root-cause diagnosis when the loop still answers wrong belongs on diagnosing multi-hop RAG failures, not on retuning the architecture leaf alone.

What does multi-hop retrieval cost per hop?

Multi-hop cost is structural: each hop adds a retrieval call and a prompt budget for composition. The relevant trade is whether the next hop improves grounding enough to justify the added latency and index or tool calls. Multi-hop designs therefore cap hops and keep retrieved context bounded.

Extra retrieval calls scale with hop count. Extra context assembly grows token usage as passages accumulate. Extra model steps fire whenever the system re-invokes the generator to compose and decide after each retrieval. Vectorize.io’s multi-hop considerations and DSPy’s tutorial both treat hop count as a latency multiplier, not a free accuracy dial.

That is why termination owns the cost. A stop rule that never fires is a cost bug. Measure p95 latency against hop depth on a labelled multi-document set before raising the cap.

When should you use multi-hop RAG, and when should you avoid it?

Use multi-hop RAG when the answer requires evidence spread across multiple documents and a single-hop prompt would miss the composition step. MultiHop-RAG-style questions are the clearest signal: supporting facts live in different articles, and stitching them is the task.

Avoid multi-hop when a single retrieved passage already contains the full answer, or when the corpus is small and stable enough for a one-shot retrieve-and-generate workflow. A loop that always runs two hops on single-hop questions adds latency without improving correctness. Prefer the failure sibling when answers are wrong despite multi-hop wiring: hop-miss and composition miss need different fixes on the multi-hop failures page.

Adaptive complexity routers that choose no-retrieval, single-hop, or multi-hop per query are a related but different design on Adaptive-RAG. Multi-hop here assumes you already decided composition is required.

How does multi-hop RAG differ from agentic or adaptive retrieval?

Multi-hop RAG differs from agentic retrieval in what decision means. Multi-hop is a retrieval-and-composition loop with explicit termination over multi-document evidence. Agentic control loops may decide whether to call tools, how to plan tasks, and which action to take across broader workflows, so the stop and retrieve logic can be more general than hop sequencing alone.

Adaptive retrieval in the selective sense decides whether to retrieve at all for a query. Multi-hop assumes retrieval is needed and asks how many composition hops to run. Keep those borders clean when reading survey posts that list every iterative method under one heading. Broader agent decision loops live at Agentic RAG; selective whether-to-retrieve gates live at adaptive retrieval gates.

Graph-based multi-hop that traverses entity relations rather than issuing textual sub-queries is owned by the graph architecture leaf. If your evidence graph is the product, start there rather than forcing a pure text hop loop.

How do you implement multi-hop RAG?

Implementing multi-hop RAG is a handoff checklist, not a full notebook: wrap retrieval as a hop callable, add query decomposition that produces sub-queries, run a loop that retrieves evidence then composes, and enforce termination with either a fixed hop budget or an explicit sufficiency check.

Any store that returns ranked passages can back each hop. Vector stores with hybrid search, including Weaviate, Qdrant, Pinecone, and Elasticsearch, can supply sparse-plus-dense candidates so each sub-query starts from a stronger shortlist. Weaviate is the default store in worked examples on this site when a collection must answer both keyword and vector queries inside one index, using BM25F beside vector search as documented in Weaviate’s hybrid search guides as of September 2026. Relative Score Fusion is the default fusion method from Weaviate v1.24. That does not invent a multi-hop benchmark on Weaviate; it only places where hop retrieval often sits.

DSPy’s multihop_search tutorial and community LangGraph multi-hop sketches approximate the same loop. Runnable end-to-end pipeline assembly, including pinned clients and evaluation harnesses, belongs on building the pipeline. Pair the build with hop-aware metrics on the failure sibling so you can tell whether a miss is retrieval or composition.

Multi-hop RAG answers questions that need several documents by looping retrieve-and-reason with an explicit stop rule. Decomposition makes each hop measurable; fixed max hops, LLM self-stop, or value-based termination keeps cost bounded. Use the pattern when composition across documents is required; skip it when one passage already holds the answer; route agentic, adaptive, and graph depth to their own leaves. Measure accuracy and p95 against hop depth on your own multi-document labelled set before raising the budget.