Multi-Hop RAG: Answering Questions That Need Several Documents

Multi-Hop RAG is an iterative retrieve-and-reason architecture for questions that need evidence spread across multiple documents. Instead of one retrieve-then-generate pass, multi-hop designs fetch evidence in multiple hops and stop the loop only when the answer is sufficiently grounded. The core mechanism is the same retrieve → reason → retrieve progression that “iterative retrieval” guides describe, but with explicit termination so the loop cannot spin.

How does multi-hop RAG work?

Multi-hop RAG is a loop where each hop retrieves evidence for the next composition step and then the system decides what to do next.

1. Retrieve for the current hop. Pull candidate passages for the current sub-question.
2. Reason from retrieved evidence. Compose or refine the partial answer that this hop enables.
3. Decide whether another hop is needed. Use the termination rule to either stop or continue.
4. Repeat with updated sub-queries. On the next hop, retrieve what the next reasoning step depends on.

The loop stays grounded because the model’s next step is conditioned on what it actually retrieved in the previous hop.

How does query decomposition help multi-hop retrieval?

Query decomposition improves multi-hop retrieval by turning one question into a sequence of smaller sub-questions so each hop can retrieve the specific evidence piece a later reasoning step depends on.

• It reduces retrieval ambiguity. Each sub-query targets a single evidence type or document requirement rather than trying to retrieve the whole answer in one shot.
• It makes hop boundaries explicit. You can log which sub-query produced which retrieved evidence and how the composition step used it.
• It supports bounded loops. Decomposition pairs naturally with max hops and sufficiency checks.

If you want the dedicated mechanism page for the first step, see query decomposition.

When should multi-hop retrieval stop?

Stop criterion is the difference between a helpful multi-hop loop and an expensive one. In practice you choose among three termination regimes.

• Fixed max hops. Cap the loop by budget so the worst-case latency stays predictable.
• LLM self-stop. Ask the model to decide sufficiency based on retrieved evidence and the partial answer state. Use it only when you trust the model’s ability to evaluate its own evidence coverage.
• Value based termination. Use a learned or scripted stop decision, like Stop-RAG style objectives, that predicts whether another hop improves the answer enough to justify the cost.

Multi-hop failure diagnosis lives on the failure sibling, since hop-miss versus composition miss needs different fixes: multi-hop failure diagnosis.

What does multi-hop retrieval cost per hop?

Multi-hop retrieval adds cost per hop because every loop iteration repeats both retrieval work and generation work. The practical trade is whether the added hop improves grounding enough to justify the extra latency and token budget.

• Extra retrieval calls. Each hop queries the index or retriever again, so index calls scale with hop count.
• Extra context assembly. Retrieved passages must be added to the prompt, so token usage grows with the number of hops you allow.
• Extra model steps. If the system re-invokes the model to compose and decide after each retrieval, the compute cost scales with the loop length.

That is why multi-hop designs enforce caps and bounded context windows: termination owns the cost.

When should you use multi-hop RAG, and when should you avoid it?

Multi-hop RAG is worth it when the answer requires multi-document composition. It is usually overkill when one retrieved passage already contains the complete answer or when the corpus is small and stable enough for a single-hop workflow.

• Use multi-hop when composition is required. The model needs evidence from multiple documents to answer correctly.
• Avoid multi-hop when one-hop grounding suffices. If your gold answer already appears in one chunk or one supporting document, a loop adds latency without improving correctness.
• Prefer the right sibling when things fail. If retrieval fails to fetch the required evidence, the fix belongs to multi-hop failure diagnosis.

How does multi-hop RAG differ from agentic or adaptive retrieval?

Multi-hop and agentic retrieval both iterate, but they do not iterate for the same reason. Multi-hop is primarily a retrieval-and-composition loop over multiple documents with explicit termination. Agentic control loops can be more general: they may decide whether to call tools, how to plan tasks, and which action to take beyond hop sequencing.

For the broader control-loop family, see Agentic RAG.

How do you implement multi-hop RAG?

Implementing multi-hop RAG is a handoff checklist: wrap retrieval so it can be called per hop; generate decomposed sub-queries; run the retrieve-and-reason loop; and enforce termination with a budget or sufficiency check.

1. Wrap retrieval as a hop callable. Treat retrieval as a function the loop can call for the current sub-query.
2. Generate sub-queries. Use query decomposition so each hop retrieves evidence for one reasoning dependency.
3. Run the retrieve → reason → decide loop. After each hop, update the partial answer state and evaluate whether you are done.
4. Enforce termination. Stop at fixed max hops or via a sufficiency check.

For runnable pipeline code, use the build path in RAG pipeline build.