Why RAG Can’t Answer Questions That Need Two Documents

A multi-hop RAG failure is when the correct answer requires chaining evidence from two or more documents, but a single retrieve-then-generate pass returns an incomplete or wrong answer. That fault is different from a wrong chunk of a one-document answer, and different from retrieving nothing at all, which is a missing document. Tang and Yang’s MultiHop-RAG benchmark (COLM 2024; arXiv:2401.15391) put 2,556 such queries against a news knowledge base, each with supporting evidence spread across 2 to 4 documents, and found existing single-shot RAG inadequate on both retrieval and answering. The RAG systems fail taxonomy places this leaf among composition faults: inspect the retrieved set against the gold evidence set for the failing query, then name whether the second document never entered the context or the documents were present and the model still failed to compose them.

What is a multi-hop question in RAG?

A multi-hop question in RAG is a query whose answer cannot be derived from a single piece of supporting evidence: the system must retrieve and reason over multiple chunks, usually from multiple documents. Tang and Yang (2024) contrast that with a single-hop question such as “What is Google’s profit margin in the third-quarter reports for 2023?”, where one evidence span is enough.

Their MultiHop-RAG dataset labels four query types (Table 3): inference (816 queries, 31.92%), comparison (856, 33.49%), temporal (583, 22.81%), and null (301, 11.78%). Null queries have no answerable evidence and test whether the model refuses instead of hallucinating. Evidence-count mix (Table 4): 42.18% of queries need two pieces of evidence, 30.48% need three, and 15.56% need four. A canonical comparison example from the paper (and Cobus Greyling’s 2024 write-up of it) asks which company among Google, Apple, and Nvidia reported the largest profit margins in their third-quarter reports for 2023: three reports, one comparison. The iterative retrieve-and-reason designs that answer that class of question live under iterative multi-hop retrieval; the diagnosis of why single-shot pipelines miss them belongs with the failure leaf.

Once the question type is clear, the next job is to prove the failure is multi-hop at all, rather than a single-document retrieval miss dressed up as a composition problem.

First, is it multi-hop or just a wrong chunk?

A failing multi-document answer has three mutually exclusive signatures. Most debugging guides either report aggregate Hits@K on MultiHop-RAG or list “multi-hop” as one bullet among ten failure modes. The faster path is a per-query branch: label the gold evidence document ids G, take the retrieved ids R at production k, and compute evidence_recall = |G ∩ R| / |G|. That single ratio rules the causes in or out before any architecture change.

Competitors publish MultiHop-RAG aggregate Hits@K; a per-query evidence_recall branch isolates the failure mode on one query. Read the differential as a triage card: each row is a pass/fail on labelled gold evidence, not a ranking of how often each cause appears on your corpus. Fleet meters such as Hits@10 still matter for overall health, but they do not name whether this query lost a hop or failed to compose.

When evidence_recall sits strictly between 0 and 1, the second document never entered the prompt, and that hop-miss signature is what single-shot retrieval produces when Doc B never ranks.

Why does single-shot retrieval miss the second document?

Single-shot retrieval misses the second document because similarity search ranks chunks against the query string, not against the information the first hit implies, so Doc A can enter top-k while Doc B, which the answer also needs, never does. Naive retrieve-then-generate runs one similarity query. Dense and sparse retrievers optimise lexical or semantic closeness to that query; they do not plan a second hop from the first evidence.

Tang and Yang (2024) ran LlamaIndex RAG with 256-token chunks: even voyage-02 plus bge-reranker-large reached only Hits@10 0.7467, and Hits@4 fell to 0.6625 (Table 5). Production context windows usually keep k in that low range, so the missing hop is cut off before generation starts. HopRAG (Liu et al., ACL Findings 2025) reports BGE recall saturating near 0.45 on MuSiQue, 2WikiMultiHopQA and HotpotQA, with over 60% of retrieved passages only indirectly relevant. Hybrid fusion in Weaviate, Pinecone, Qdrant and Milvus can raise first-hop lexical recall when the query shares tokens with Doc A, but fusion still scores one query against the corpus; it does not invent the second hop. Weaviate’s hybrid search documentation (as of September 2026) defines hybrid as combining vector search with keyword BM25F search, then fusing the two result sets for that single query string.

When every gold evidence id is already in the prompt and the answer is still wrong, the failure has moved off retrieval and into composition, which is a different measurement and a different fix.

Why is the answer still wrong when all gold evidence is in the context?

When every gold evidence chunk is already in the prompt and the answer is still wrong, the failure is composition: the model did not chain the facts, even though retrieval succeeded. Multi-hop answering needs relational reasoning across evidence, not just presence of the right passages.

Tang and Yang (2024, Table 6) measured the gap directly. With the best retrieved chunks, GPT-4 accuracy was 0.56; with ground-truth evidence it rose to 0.89, still not perfect. On gold evidence, Llama-2-70B reached 0.32 and Mixtral-8x7B-Instruct 0.36. Kuldeep Paul’s production failure catalogue (2025) names the same pattern: systems retrieve the necessary facts and still fail multi-hop inference. Barnett et al. (2024) list incomplete multi-document answers as failure point 7 even when the missing facts were in context.

Before changing either retrieval or reasoning, the practical question is how to detect that a production query needs more than one hop in the first place.

How do you detect that a query needs more than one hop?

You detect a multi-hop requirement by labelling the gold evidence set and measuring whether production retrieval covers it, not by reading the answer’s confidence. Kuldeep Paul (2025) recommends evaluation sets where ground truth names which documents hold which reasoning steps. Tang and Yang (2024, §2.3) define Hit@K as the fraction of gold evidence that appears in the top-K retrieved set, which is evidence_recall under another name.

Four practical checks run in order. First, label G: for each failing query, record the document or chunk ids a human needs to answer it. Building that set is golden test set work; Hits and MAP definitions sit on retrieval metrics. Second, score evidence_recall at production k: partial coverage means hop-miss, full coverage with a wrong answer means composition miss, and empty coverage with a one-document gold means the fault is not multi-hop. Third, apply the bridge test: if two facts only join through a shared entity or topic (Tang and Yang’s bridge-entity / bridge-topic), treat the query as multi-hop even when each fact alone looks easy. Fourth, rewrite as single-hop sub-queries: split into “What is X?” and “What is Y?”; if each sub-query retrieves its gold and a composed answer works, the original question was multi-hop.

1. Label G. Record the evidence ids a competent reader needs. Without that label set, aggregate answer accuracy cannot separate hop-miss from composition miss.
2. Score evidence_recall at production k. Map the ratio onto the three signatures in the differential table.
3. Bridge test. Shared entity or topic between otherwise separate facts is a multi-hop signal even when each fact looks single-hop alone.
4. Single-hop rewrite. Successful composed answers from split sub-queries confirm the original was multi-hop.

Log the query, retrieved ids, similarity scores, and which gold ids hit, using the same trace you would keep for RAG tracing. With the cause named, the remaining question is which architecture class fixes hop-miss versus composition miss without turning this diagnosis leaf into a build tutorial.

What architectures fix multi-hop RAG failures?

Multi-hop RAG failures are fixed by architectures that retrieve more than once or reason over structured links, not by a larger top-k on the same single query. Depth for each design lives on its own node; the routes that follow name the cause and the owning mechanism page so the failure leaf stays a diagnosis chapter.

Match the cause before picking a stack. Hop-miss needs another retrieval step. Composition miss needs better reason-and-verify even when evidence_recall is already 1. Tang and Yang (2024, §4.3) flag query decomposition and LLM agents as the natural next experiments after their naive LlamaIndex baseline; they do not claim a single winner, and neither does this routing list.

• Query decomposition: split the compound question into single-document sub-queries, retrieve for each, then compose. Primary lever for hop-miss. Mechanism: splitting multi-part questions.
• Multi-query and fusion: ask several related queries and merge ranked lists when the cheaper fan-out is enough before a full iterative loop. Mechanism: asking several queries and fusing ranks.
• Iterative multi-hop retrieval: retrieve, reason, retrieve again, with an explicit stopping criterion so the loop cannot spin. Primary fix target for this failure: multi-hop RAG.
• Agentic control: the model decides whether, what, and how many times to retrieve when hop count is unpredictable. Mechanism: agentic control loops.
• Graph / logic-aware hops: passages linked by relations or pseudo-queries (HopRAG’s retrieve-reason-prune; GraphRAG) when the corpus has stable entities and relations. Mechanism: GraphRAG.
• IRCoT: interleave chain-of-thought with retrieval so each thought can fetch the next fact. Mechanism: IRCoT.

How-to depth for decomposition, stopping criteria, and agent loops is owned by those mechanism pages. What remains here is measuring the failure on a labelled eval set so the cause name is reproducible after each deploy.

How do you measure multi-hop failures on an eval set?

You measure multi-hop failures by scoring evidence-set coverage and answer correctness together, then naming the cause, not by answer accuracy alone. The function below takes a labelled gold evidence set and the ids your retriever returned; it prints hop-miss, composition-miss, or not-multi-hop. Wire the same checks into CI on regression testing once the labels live in a labelled golden set. Public sets such as MultiHop-RAG and HotpotQA are catalogued under RAG benchmarks.

Each branch of the script matches one row of the differential table. Delete the causes you have already ruled out; keep the labelled gold_ids, because without them the classifier cannot distinguish a hop-miss from a composition miss. MultiHop-RAG’s Hits@10 0.7467 and GPT-4 accuracies 0.56 / 0.89 (Tang and Yang, 2024) are news-domain ceilings on that benchmark, not universal constants, so calibrate thresholds on your own labelled multi-hop slice before you change architecture.

def diagnose_multi_hop(gold_ids, retrieved_ids, *, answer_correct, k=None):
    """Name hop-miss vs composition-miss vs not-multi-hop for one labelled query.

    gold_ids       -> iterable of evidence document/chunk ids required for the answer
    retrieved_ids  -> ranked list from your retriever (production top-k, or wider for debug)
    answer_correct -> True if the generated answer matches your gold label
    k              -> optional cutoff; defaults to len(retrieved_ids)
    """
    G = set(gold_ids)
    if not G:
        print("NO GOLD EVIDENCE labelled; cannot diagnose multi-hop")
        return

    R = list(retrieved_ids) if k is None else list(retrieved_ids)[:k]
    R_set = set(R)
    hit = G & R_set
    evidence_recall = len(hit) / len(G)

    print(f"evidence_recall={evidence_recall:.2f} "
          f"(hit {len(hit)}/{len(G)} gold ids in top-{len(R)})")

    if len(G) == 1 and evidence_recall == 0:
        print("CAUSE not-multi-hop -> see /failures/wrong-chunk or /failures/missing-document")
        return
    if evidence_recall == 0:
        print("CAUSE not-multi-hop or total miss -> inspect indexing; not a hop-composition case yet")
        return
    if evidence_recall < 1:
        missing = sorted(G - R_set)
        print(f"CAUSE hop-miss (missing {missing}) -> multi-hop retrieval / query decomposition")
        return
    if not answer_correct:
        print("CAUSE composition-miss (all gold evidence present) -> multi-hop / agentic reason+verify")
        return
    print("PASS evidence complete and answer correct")

Single-shot RAG fails multi-document questions for two separable reasons: the second document never enters top-k, or every gold passage is present and composition still fails. Evidence_recall names which cause fired on a labelled query, and that name decides whether the next change is another retrieval step or stronger reason-and-verify. Until that measurement exists, swapping embeddings or raising k alone treats a hop composition fault as if it were an ordinary single-document ranking miss.