Why RAG Can’t Answer Questions That Need Two Documents

A multi-hop RAG failure is when the correct answer requires chaining evidence from two or more documents, but a single retrieve-then-generate pass returns an incomplete or wrong answer. That is different from a wrong chunk of a one-document answer, and different from retrieving nothing at all (missing document). Tang and Yang’s MultiHop-RAG benchmark (COLM 2024; arXiv:2401.15391) put 2,556 such queries against a news knowledge base — each with supporting evidence spread across 2 to 4 documents — and found existing single-shot RAG inadequate on both retrieval and answering.

Before you rebuild the stack as “agentic” or “graph,” inspect the retrieved set against the gold evidence set for the failing query. The question is not whether RAG “supports multi-hop” in the abstract — it is which of two causes fired on this query: the second document never entered the context, or the documents were there and the model still failed to compose them.

What is a multi-hop question in RAG?

A multi-hop question in RAG is a query whose answer cannot be derived from a single piece of supporting evidence — the system must retrieve and reason over multiple chunks, usually from multiple documents. Tang and Yang (2024) contrast that with a single-hop question such as “What is Google’s profit margin in the third-quarter reports for 2023?”, where one evidence span is enough.

Their MultiHop-RAG dataset labels four query types (Table 3): inference (816 queries, 31.92%), comparison (856, 33.49%), temporal (583, 22.81%), and null (301, 11.78%) — null queries have no answerable evidence and test whether the model refuses instead of hallucinating. Evidence-count mix (Table 4): 42.18% of queries need two pieces of evidence, 30.48% need three, and 15.56% need four. A canonical comparison example from the paper (and Cobus Greyling’s 2024 write-up of it): “Which company among Google, Apple, and Nvidia reported the largest profit margins in their third-quarter reports for 2023?” — three reports, one comparison. The fix architectures for that class of question live on multi-hop RAG; this page is the failure diagnosis.

First, is it multi-hop or just a wrong chunk?

A failing multi-document answer has three mutually exclusive signatures. Most debugging guides either report aggregate Hits@K on MultiHop-RAG or list “multi-hop” as one bullet among ten failure modes. The faster path is a per-query branch: label the gold evidence document ids G, take the retrieved ids R at production k, and compute evidence_recall = |G ∩ R| / |G|. That single ratio rules the causes in or out before you change code.

Why does single-shot retrieval miss the second document?

Single-shot retrieval misses the second document because similarity search ranks chunks against the query, not against the information the first hit implies — so Doc A can enter top-k while Doc B, which the answer also needs, never does.

The gold evidence is all in the context — why is the answer still wrong?

When every gold evidence chunk is already in the prompt and the answer is still wrong, the failure is composition — the model did not chain the facts — not retrieval.

How do you detect that a query needs more than one hop?

You detect a multi-hop requirement by labelling the gold evidence set and measuring whether production retrieval covers it — not by reading the answer’s confidence. Kuldeep Paul (2025) recommends evaluation sets where ground truth names which documents hold which reasoning steps; Tang and Yang (2024, §2.3) define Hit@K as the fraction of gold evidence that appears in the top-K retrieved set — that is evidence_recall under another name.

Four practical checks, in order:

1. Label G. For each failing query, record the document (or chunk) ids a human needs to answer it. Building that set is golden test set work; Hits/MAP definitions sit on retrieval metrics.
2. Score evidence_recall at production k. Partial coverage → hop-miss; full coverage with a wrong answer → composition miss; empty coverage with a one-document gold → not multi-hop.
3. Bridge test. If two facts only join through a shared entity or topic (Tang and Yang’s bridge-entity / bridge-topic), treat the query as multi-hop even when each fact alone looks easy.
4. Single-hop rewrite. Split into “What is X?” and “What is Y?”; if each sub-query retrieves its gold and a composed answer works, the original question was multi-hop.

Log the query, retrieved ids, similarity scores, and which gold ids hit — the same trace you would use on RAG tracing. Without G labelled, aggregate answer accuracy cannot tell hop-miss from composition miss.

What architectures fix multi-hop RAG failures?

Multi-hop RAG failures are fixed by architectures that retrieve more than once or reason over structured links — not by a larger top-k on the same single query. Depth for each design lives on its own node; this section only routes by cause.

• Query decomposition — split the compound question into single-document sub-queries, retrieve for each, then compose. Primary lever for hop-miss. See query decomposition.
• Iterative multi-hop retrieval — retrieve, reason, retrieve again, with an explicit stopping criterion so the loop cannot spin. Primary page for this failure’s fix: multi-hop RAG.
• Agentic control — the model decides whether, what, and how many times to retrieve. Use when hop count is unpredictable. See agentic RAG.
• Graph / logic-aware hops — passages linked by relations or pseudo-queries (HopRAG’s retrieve-reason-prune; GraphRAG). Use when the corpus has stable entities and relations. See GraphRAG.
• IRCoT — interleave chain-of-thought with retrieval so each thought can fetch the next fact. See IRCoT.

Rule of thumb: hop-miss needs another retrieval step; composition miss needs better reason-and-verify even when evidence_recall is already 1. Tang and Yang (2024, §4.3) flag query decomposition and LLM agents as the natural next experiments after their naive LlamaIndex baseline — they do not claim a single winner.

How do you measure multi-hop failures on an eval set?

You measure multi-hop failures by scoring evidence-set coverage and answer correctness together, then naming the cause — not by answer accuracy alone. The function below takes a labelled gold evidence set and the ids your retriever returned; it prints hop-miss, composition-miss, or not-multi-hop. Wire the same checks into CI on regression testing once the labels live in a golden test set. Public sets such as MultiHop-RAG and HotpotQA are catalogued under RAG benchmarks.

def diagnose_multi_hop(gold_ids, retrieved_ids, *, answer_correct, k=None):
    """Name hop-miss vs composition-miss vs not-multi-hop for one labelled query.

    gold_ids      -> iterable of evidence document/chunk ids required for the answer
    retrieved_ids -> ranked list from your retriever (production top-k, or wider for debug)
    answer_correct -> True if the generated answer matches your gold label
    k             -> optional cutoff; defaults to len(retrieved_ids)
    """
    G = set(gold_ids)
    if not G:
        print("NO GOLD EVIDENCE labelled — cannot diagnose multi-hop")
        return

    R = list(retrieved_ids) if k is None else list(retrieved_ids)[:k]
    R_set = set(R)
    hit = G & R_set
    evidence_recall = len(hit) / len(G)

    print(f"evidence_recall={evidence_recall:.2f} "
          f"(hit {len(hit)}/{len(G)} gold ids in top-{len(R)})")

    if len(G) == 1 and evidence_recall == 0:
        print("CAUSE not-multi-hop -> see /failures/wrong-chunk or /failures/missing-document")
        return
    if evidence_recall == 0:
        print("CAUSE not-multi-hop or total miss -> inspect indexing; not a hop-composition case yet")
        return
    if evidence_recall < 1:
        missing = sorted(G - R_set)
        print(f"CAUSE hop-miss (missing {missing}) -> multi-hop retrieval / query decomposition")
        return
    if not answer_correct:
        print("CAUSE composition-miss (all gold evidence present) -> multi-hop / agentic reason+verify")
        return
    print("PASS evidence complete and answer correct")

Each branch matches one row of the differential table above. Delete the causes you have already ruled out; keep the labelled gold_ids — without them the script cannot distinguish a hop-miss from a composition miss.