How to Find Which Stage of Your RAG Pipeline Broke

A broken RAG answer almost always fails in one stage of the pipeline — index, embedding, retrieval, context assembly, or generation — and the fastest path is to isolate that stage with one pass/fail measurement before you change any code. This page is the diagnostic spine: it names which stage broke. Once you know the stage, the matching failure-mode page owns the cause and the fix. Organisational blockers (permissions, source sprawl, no success metric) are a different class of failure — see why RAG pilots fail in enterprises.

What are the stages that can break in a RAG pipeline?

A RAG pipeline has five diagnostic stages for isolation: index presence, embedding match, retrieval, context assembly, and generation. Index-time failures show up as missing or stale candidates; query-time failures show up as wrong ranking or an answer that ignores good context. Building those stages from scratch is covered under how to build a RAG pipeline — this page only measures which one broke.

Practitioners often report that most RAG failures happen before the LLM call (retrieval and context assembly). Treat that as orientation from field guides such as Sindhu Murthy’s production troubleshooting walkthrough (dev.to), not as a published rate with a methodology you can cite (REB-26).

How do you isolate retrieval failure from generation failure?

Force-feed the model the gold context for the failing query. If the answer becomes correct with gold context, retrieval (or an earlier stage) failed; if it stays wrong with gold context, generation failed. That single A/B is the retrieval-versus-generation gate Zen van Riel’s debugging guide opens with — run it before you swap models or rewrite prompts.

When the gold-context A/B says retrieval failed, open why your RAG returns the wrong chunk. When gold context still fails, open lost in the middle or hallucination despite context.

What measurement proves the document never entered the index?

Search the store for a unique string you know is in the source document — keyword/BM25 search or the database admin UI. If that string is absent from every indexed chunk, the failure is index or ingest, not the LLM. The measurement is binary: corpus_hit = 1 if the gold string appears in any chunk, else 0.

A zero here routes to when RAG misses the document you know is there. In enterprises, the usual reason the string never landed is a missing connector or sync job — see connecting RAG to Drive, Notion and SharePoint.

How do you check the query embedding matches the index?

Log the embedding model id used at query time and compare it to the model id that built the index. A mismatch places queries and documents in different vector spaces and returns confidently irrelevant neighbours with no exception thrown — Murthy’s “silent killer.” The measurement is embed_model_match = (query_model_id == index_model_id).

If the check fails, re-embed the corpus with the query model or restore the query model to the index model before you tune retrieval. Ongoing mismatch after model upgrades is embedding drift; deleted or replaced docs that still answer are a stale index.

How do you measure whether retrieval returned the right chunks?

For a labelled failing query with a known gold chunk id, compute recall@k: 1 if the gold id is in the top-k returned ids, else 0. If recall@k is 0 after index and embedding already passed, the broken stage is retrieval — do not open the LLM logs yet.

What recall@k means as a metric family is on measuring retrieval. Why retrieval missed the gold chunk — vocabulary, granularity, ranking, embedding geometry, or filters — is the five-cause page at wrong chunk.

How do you check whether the gold chunk reached the prompt?

After retrieval passes, assert that the gold chunk text (or id) appears in the assembled prompt string and that the context was not truncated. The measurement is gold_in_prompt = 1 if the gold substring is in the final prompt, else 0. Failures here are assembly, budget, or ordering problems — not “the model is dumb.”

When the gold chunk was present in the prompt and still ignored, that is lost in the middle.

How do you measure whether generation stayed grounded?

With retrieval and prompt assembly already passing, score faithfulness (groundedness) of the answer against the retrieved context — or reuse the gold-context A/B from the isolation section. If faithfulness fails while correct context is in the prompt, the broken stage is generation.

How to score generation is on generation metrics. The residual hallucination problem when context was fine is does RAG fix hallucination?.

How do you run the stage checks in order?

Run five checks in order on one failing query and its gold chunk id; stop at the first fail — that names the stage. You need a tiny labelled set first (even ten failing queries is enough to start) — see golden test sets. In production, bind every stage log to the same request_id so the walk is one trace (tracing a RAG pipeline).

def isolate_rag_stage(
    *,
    gold_string: str,
    gold_id: str,
    indexed_chunks: list[dict],          # [{"id": ..., "text": ...}, ...]
    query_embed_model: str,
    index_embed_model: str,
    retrieved_ids: list[str],            # top-k ids for this query
    final_prompt: str,
    answer_with_retrieved: str,
    answer_with_gold_context: str,
    judge_ok,                            # callable(answer, context) -> bool
):
    """Return the first failing stage name for one labelled query.

    Stop at the first False. Do not change pipeline code until this returns.
    """
    corpus_hit = any(gold_string in c["text"] for c in indexed_chunks)
    if not corpus_hit:
        return "index"  # document never entered (or wrong collection)

    if query_embed_model != index_embed_model:
        return "embed"  # vector spaces diverge — every similarity score is noise

    if gold_id not in retrieved_ids:
        return "retrieve"  # recall@k = 0

    if gold_string not in final_prompt and gold_id not in final_prompt:
        return "prompt"  # retrieved but dropped/truncated in assembly

    if judge_ok(answer_with_gold_context, gold_string) and not judge_ok(
        answer_with_retrieved, final_prompt
    ):
        return "retrieve-or-prompt"  # gold context works → earlier stage lied

    if not judge_ok(answer_with_gold_context, gold_string):
        return "generate"  # still wrong with perfect context

    return "pass"  # this query's failure is elsewhere (permissions, stale source, …)

# Minimal judge for demos: answer must share a distinctive gold token.
# Replace with a faithfulness scorer in production (see /evaluation/generation-metrics/).
def contains_gold(answer: str, context: str) -> bool:
    token = context.split()[0]
    return token.lower() in answer.lower()

Catalogs of every failure mode (chunking, hybrid, rerank, stale indexes) are useful after isolation — they are the rest of the failures hub, not a substitute for naming the stage. Debugging-tool surveys belong under evaluation tools and production monitoring.