How to Find Which Stage of Your RAG Pipeline Broke

A broken RAG answer almost always fails in one stage of the pipeline: index presence, embedding match, retrieval, context assembly, or generation. The fastest fix path is to isolate that stage with one pass/fail measurement before you change any code. The diagnostic spine is stage naming, not cause hunting: once the stage is known, the matching leaf under failures sorted by the stage that caused it owns the mechanism and the repair. Latitude’s July 2026 debugging guide opens with the same order (inspect retrieval, then test generation with known-good context), and Autonoma’s July 2026 testing write-up splits the same fault into two surfaces that end-to-end scores cannot localize. Organisational blockers such as permissions sprawl or a missing success metric are a different class of failure; those route to why RAG pilots fail in enterprises.

What are the stages that can break in a RAG pipeline?

A RAG pipeline has five diagnostic stages for isolation: index presence, embedding match, retrieval, context assembly, and generation. Index-time failures surface as missing or stale candidates; query-time failures surface as wrong ranking or an answer that ignores good context. Progress’s July 2026 pipeline explainer (Adam Bertram) walks a related four-stage build path of ingestion, parsing, chunking, and retrieval orchestration, and LinkedIn’s four-stage deep dive by Aanchal Sikarwar frames the same machinery as the path a document travels before an answer exists. Diagnostic stages collapse that machinery into measurements you can pass or fail on one labelled query.

Building those stages from scratch is covered under how to build a RAG pipeline. Isolation only measures which one broke. Practitioner guides often claim that most RAG failures happen before the LLM call. Latitude’s July 2026 post treats the circulating “70% are retrieval” line as a widely repeated heuristic with no primary source behind it, and notes that the same figure floats around as 60% elsewhere. Treat that direction as orientation, not as a published rate with a methodology you can cite. Barnett et al. (arXiv 2401.05856, 2024) land on a sharper production claim: a RAG system’s robustness “evolves rather than [is] designed in at the start,” and validation “is only feasible during operation,” which is why stage checks belong on live failing queries, not only on a demo notebook.

Once the stage map is clear, the next cut is coarser and faster than walking every gate: force the model to answer from gold context and see whether the fault sits on the retrieval side or the generation side.

How do you isolate retrieval failure from generation failure?

Force-feed the model the gold context for the failing query. If the answer becomes correct with gold context, retrieval or an earlier stage failed; if it stays wrong with gold context, generation failed. That single A/B is the retrieval-versus-generation gate Latitude’s July 2026 guide calls the cleanest fault-localization test in RAG, and Autonoma’s July 2026 article makes the same split the spine of its two-surface test design. Run it before you swap models, rewrite prompts, or retune top-k.

Arthur.ai’s RAG-experiment framing explains why the gate matters at agent level: bad retrieval produces a confidently wrong answer with no error signal, because the agent reasoned over whatever context it received. Autonoma states the two failure signatures in operator language. Retrieval failure is right question, wrong chunks, with the fix in chunking, embeddings, top-k, or reranking. Generation failure is right chunks, unsupported claim, with the fix in the prompt, faithfulness threshold, or refusal logic. An end-to-end wrong-answer score makes those two look identical from the outside.

When the gold-context A/B says retrieval failed, open why your RAG returns the wrong chunk. When gold context still fails, open why models ignore mid-context evidence or does RAG fix hallucination?. The A/B halves the search space; the remaining gates name which earlier stage lied when retrieval looked guilty.

What measurement proves the document never entered the index?

Search the store for a unique string you know is in the source document, using keyword or BM25 search or the database admin UI. If that string is absent from every indexed chunk, the failure is index or ingest, not the LLM. The measurement is binary: corpus_hit = 1 if the gold string appears in any chunk, else 0. Latitude’s first inspection step asks the same question in plain language: is the answer even in your corpus? Progress’s ingestion stage makes the same point from the other direction: if last quarter’s pricing exception sits in an email thread no connector touches, no query can recover it.

On a store that supports keyword search, the cheapest proof is an exact-token lookup rather than a similarity query, because you are testing presence, not rank. Weaviate’s keyword search documentation (as of September 2026) describes BM25F as the sparse path that returns objects with the highest BM25F scores for a search string, which is the right tool for a distinctive gold phrase. Popular vector stores for the same check include Weaviate, Pinecone, Qdrant, and Milvus; the measurement stays the same across products even when the API name differs.

A zero here routes to when RAG misses the document you know is there. In enterprises, the usual reason the string never landed is a missing connector or sync job; see connecting RAG to Drive, Notion and SharePoint. If the string is present but every later gate still fails, the fault has moved into embedding or retrieval, not ingest.

How do you check the query embedding matches the index?

Log the embedding model id used at query time and compare it to the model id that built the index. A mismatch places queries and documents in different vector spaces and returns confidently irrelevant neighbours with no exception thrown. Latitude’s symptom table lists the same failure as config or embedding-model drift after a code or dependency change, and the fix is explicit: pin the embedding model name and version; if you change it, rebuild the whole index. The measurement is embed_model_match = (query_model_id == index_model_id).

Arthur.ai treats embedding model as one of the four variables a RAG experiment should change one at a time, because a model that ranks well on a general leaderboard can miss domain matches. Atlan’s May 2026 accuracy survey cites the FinMTEB finding that general MTEB rankings correlate poorly with financial-domain retrieval, which is the same class of silent mismatch: the pipeline runs, the scores look numeric, and the neighbours are wrong for the corpus you actually have.

If the check fails, re-embed the corpus with the query model or restore the query model to the index model before you tune retrieval. Ongoing mismatch after model upgrades is embedding drift; deleted or replaced docs that still answer are a stale index. Do not raise top-k or add a reranker while this bit is false: every similarity score is noise until the spaces match.

How do you measure whether retrieval returned the right chunks?

For a labelled failing query with a known gold chunk id, compute recall@k: 1 if the gold id is in the top-k returned ids, else 0. If recall@k is 0 after index and embedding already passed, the broken stage is retrieval. Do not open the LLM logs yet. Autonoma’s retrieval surface states the same assertion: for a question with a known-correct chunk, is that chunk in the top-k, and at what rank. Rank matters because a chunk at position 15 when you only pass the top 5 is, functionally, not retrieved, which Latitude lists as a separate inspection question after “did retrieval return the relevant chunks at all.”

Vishal Mysore’s Medium pipeline guide (visrow) defines Precision@K and Recall@K in classic IR terms and places them before generation metrics for the same reason: retrieval is the deterministic half of the path given a fixed index. Arthur.ai prefers binary, specific, explanation-backed evals over 1-to-10 scores for the same gate, and recommends programmatic checks for quantitative measures such as context recall rather than forcing an LLM judge where a set membership test is enough. What recall@k means as a metric family is on measuring retrieval with recall.

Why retrieval missed the gold chunk (vocabulary, granularity, ranking, embedding geometry, or filters) is the five-cause page at wrong-chunk retrieval failures. The stage gate only answers whether the gold id entered the candidate set; the sibling owns the cause once that bit is zero.

How do you check whether the gold chunk reached the prompt?

After retrieval passes, assert that the gold chunk text or id appears in the assembled prompt string and that the context was not truncated. The measurement is gold_in_prompt = 1 if the gold substring is in the final prompt, else 0. Failures here are assembly, budget, or ordering problems, not “the model is dumb.” Latitude lists context assembly as its own wrong-answer stage: the right chunk was fetched but then truncated, buried in the middle of a long context, or stripped of the attribution the model needed.

Log the final prompt token count beside a context_truncated flag whenever your stack exposes one. A gold id that appears in retrieved ids but not in the prompt string is the clearest assembly failure signature. When the gold chunk was present in the prompt and still ignored, that is lost-in-the-middle position bias, which Liu et al. (TACL 2023 / arXiv 2307.03172) measured as a U-shaped use-of-context curve: models use information best at the beginning or end of the context and worse in the middle. Reordering and cutting k are the next levers once presence is proven; they are not substitutes for proving presence first.

If gold_in_prompt fails, fix assembly and budget before you change the generator. If it passes, the remaining question is whether generation stayed faithful to the context that actually arrived.

How do you measure whether generation stayed grounded?

With retrieval and prompt assembly already passing, score faithfulness or groundedness of the answer against the retrieved context, or reuse the gold-context A/B from the isolation section. If faithfulness fails while correct context is in the prompt, the broken stage is generation. Autonoma’s generation surface weights faithfulness (every claim supported by context) and answer relevancy (the answer actually addresses the question), and adds a third case: questions the corpus cannot answer should refuse or hedge rather than invent. Progress’s REMi evaluation layer names Groundedness for the same signal: whether the retrieved context supports the answer.

Es and James introduced the RAGAS metric family (EACL 2024 / arXiv 2309.15217) specifically so faithfulness, answer relevancy, context precision, and context recall can target distinct failure modes. Atlan’s May 2026 survey presents that four-metric table as the industry-standard measurement set and attributes high monthly evaluation volume to the RAGAS project; treat the volume claim as Atlan’s reporting, and treat the metric definitions as the paper’s. How to score generation in detail is on generation groundedness metrics. The residual hallucination problem when context was fine is hallucination despite retrieved context.

Prefer the gold-context A/B when you lack a judged faithfulness metric, because a binary “gold context works / gold context fails” still names the stage without inventing a threshold. Once generation is the named stage, prompt constraints, temperature, model choice, and refusal logic are in play; chunking and embeddings are not.

How do you run the stage checks in order?

Run five checks in order on one failing query and its gold chunk id; stop at the first fail, because that names the stage. You need a tiny labelled set first. Autonoma’s July 2026 guidance is that twenty to fifty curated, human-reviewed examples catch more real regressions than hundreds of unreviewed synthetic pairs; even ten failing queries is enough to start the isolation walk. Build or borrow that set under golden test sets for RAG. In production, bind every stage log to the same request_id so the walk is one trace; the mechanics of tracing a RAG pipeline make the same order faster than reproducing the bug by hand.

Each row of the stage-isolation table is one binary measurement. Tip lists and failure-mode catalogs walk causes; this sequence isolates the stage that owns the wrong answer before any code change. Romero Lauro et al. (arXiv 2504.13587, 2025) studied twelve engineers debugging RAG and found that retrieval and generation are intertwined enough that intermediate-state inspection has to come before config changes; the ordered gates below are that inspection made mechanical.

Read the table as triage, not as a ranking of how often each stage fails on your corpus. One labelled failing query, five boolean gates, one stage name. Fleet averages and tool surveys still matter later; they do not name which row fired on the request a user just pasted into Slack.

def isolate_rag_stage(
    *,
    gold_string: str,
    gold_id: str,
    indexed_chunks: list[dict],          # [{"id": ..., "text": ...}, ...]
    query_embed_model: str,
    index_embed_model: str,
    retrieved_ids: list[str],            # top-k ids for this query
    final_prompt: str,
    answer_with_retrieved: str,
    answer_with_gold_context: str,
    judge_ok,                            # callable(answer, context) -> bool
):
    """Return the first failing stage name for one labelled query.

    Stop at the first False. Do not change pipeline code until this returns.
    """
    corpus_hit = any(gold_string in c["text"] for c in indexed_chunks)
    if not corpus_hit:
        return "index"  # document never entered (or wrong collection)

    if query_embed_model != index_embed_model:
        return "embed"  # vector spaces diverge; every similarity score is noise

    if gold_id not in retrieved_ids:
        return "retrieve"  # recall@k = 0

    if gold_string not in final_prompt and gold_id not in final_prompt:
        return "prompt"  # retrieved but dropped/truncated in assembly

    if judge_ok(answer_with_gold_context, gold_string) and not judge_ok(
        answer_with_retrieved, final_prompt
    ):
        return "retrieve-or-prompt"  # gold context works; earlier stage lied

    if not judge_ok(answer_with_gold_context, gold_string):
        return "generate"  # still wrong with perfect context

    return "pass"  # this query's failure is elsewhere (permissions, stale source, ...)


# Minimal judge for demos: answer must share a distinctive gold token.
# Replace with a faithfulness scorer in production (see /evaluation/generation-metrics/).
def contains_gold(answer: str, context: str) -> bool:
    token = context.split()[0]
    return token.lower() in answer.lower()

Catalogs of every failure mode (chunking, hybrid, rerank, stale indexes) are useful after isolation. They are the rest of the failures hub, not a substitute for naming the stage. Debugging-tool surveys belong under evaluation tools and production monitoring. When every stage gate returns pass and the answer is still wrong, the fault is usually organisational or access-shaped rather than a missing retrieval knob, which is the enterprise failure class rather than another pipeline stage.

Stage isolation turns a confidently wrong answer into a named component: index, embed, retrieve, prompt, or generate. Run the five binary checks in that order on one labelled query, stop at the first fail, and only then open the matching cause page or change code. A request-id-spanning trace makes the same walk cheap in production; a small reviewed golden set keeps the fix from silently regressing on the next deploy.