Why RAG Systems Fail: The Complete Failure Taxonomy

A RAG failure taxonomy sorts every wrong answer by the pipeline stage that caused it, because the stage, not the symptom name, decides the fix. Barnett, Kurniawan, Thudumu, Brannelly and Abdelrazek’s experience report “Seven Failure Points When Engineering a Retrieval Augmented Generation System” (arXiv 2401.05856, 11 January 2024) frames RAG failures as limits of information retrieval plus reliance on large language models, and states that validation is only feasible during operation while robustness evolves rather than being designed in at the start. Label Studio’s 19 March 2025 guide splits the same problem into a retrieval stage and a generation stage: if retrieval fails, generation always struggles. The method here is therefore stage-first: map the symptom to index and chunk, retrieve, generate, or a cross-cutting band such as freshness, security or performance, then open the leaf that owns that mode. Vector databases such as Weaviate, Qdrant and Pinecone change what retrieval can return; they do not remove the need to locate the broken stage. The seed definition of what is retrieval-augmented generation is the pipeline this taxonomy diagnoses.

What is a RAG failure taxonomy?

A RAG failure taxonomy is a map that groups wrong answers by the pipeline stage that produced them, not by how dramatic the symptom sounds. Barnett et al. (arXiv 2401.05856) publish seven failure points drawn from research, education and biomedical case studies: missing content, missing top-ranked documents, not in context, not extracted, wrong format, incorrect specificity, and incomplete answer. Label Studio (19 March 2025) and Sahin Ahmed’s Medium restatement of the same seven keep that list as the practitioner vocabulary most teardown pages still echo. A taxonomy that stops at seven named symptoms still leaves the engineer guessing whether to re-chunk, re-rank, re-prompt or rebuild permissions.

Other ranking guides sort the same mess differently. Shubham Maurya at the AI Accelerator Institute (1 October 2025) groups production pain as four horsemen: knowledge drift, retrieval decay, irrelevant chunks, and an evaluation gap. Sai Yashwanth at kapa.ai (6 January 2026) lists seven engineering mistakes: bad chunking, poor data quality, wrong architecture, no reranker, embedding rot, zero evals, and a Graph RAG dilemma. PromptQL’s failure-mode analysis on FRAMES-style questions adds extraction errors, context-size limits, inexhaustive computation and computational-reasoning errors. Those lists overlap; none of them pin every mode the topical cluster needs to a single stage and a dedicated fix page.

The taxonomy on this hub therefore uses four bands. Index and chunk failures mean the answer was mis-stored or never became a clean candidate. Retrieve failures mean the wrong passage came back, or the right one never did. Generate failures mean retrieval was fine and the answer is still wrong. Cross-cutting failures (freshness, security, performance, organisational fit) are not tied to one span but still bring the system down. Once those bands are fixed in the reader’s head, the next question is why so many teams still start by blaming the model.

Why do RAG systems fail before the model generates?

RAG systems fail before the model generates when indexing or retrieval hands the model the wrong context and the model answers that context faithfully. Label Studio’s stage split is blunt: both stages must work together, and a weak retrieval stage makes generation look broken even when the language model is behaving. Weaviate’s overview of RAG evaluation (Erika Shorten and Connor Shorten, 21 November 2023) states the same bubble-up rule for measurement: an indexing fault becomes a retrieval miss, which becomes a fluent answer that still cites the wrong passage.

Maurya’s irrelevant-chunks horseman is the overload form of that miss: the retriever returns too much noise, the model is overwhelmed, and the output starts to invent. kapa.ai’s first two mistakes, bad chunking and poor data quality, land even earlier: if documents are cut into incoherent pieces or scraped junk is embedded as-is, cosine neighbours will not save the prompt. Changing the generator temperature does not repair a corpus that never stored the answer as a retrievable unit.

That is why the common misdiagnosis is “the model hallucinated” when the retrieved set never held the fact. The leaf for a confidently wrong retrieved passage is wrong chunk retrieved. Separating retrieval scores from generation scores, so a faithful answer from the wrong document cannot paint the dashboard green, is how teams evaluate retrieval and generation separately. Once “before the model” is accepted, the next cut is retrieval versus generation when both stages look guilty from the outside.

How do retrieval failures differ from generation failures?

Retrieval failures differ from generation failures by whether the useful passage ever entered the prompt window in a usable position. Barnett et al. place Missing Content and Missing Top Ranked Documents on the retrieval side: the answer is absent from the knowledge base, or it exists and the ranker buries it below the cutoff. Generation-side points in the same paper (Not Extracted, Wrong Format, Incorrect Specificity, Incomplete Answer) assume the evidence was available and the model still misused it. Label Studio and Sahin keep that same split when they walk the seven points as a troubleshooting sequence.

On the retrieval band, five symptoms dominate production. A chunk comes back and does not contain the answer: that is why your RAG returns the wrong chunk. Nothing relevant returns even though the document exists: missing the obvious document. The user’s words do not match the document’s words: vocabulary mismatch. Structured rows collapse when tables are flattened: numbers and tables. A question needs two documents composed: multi-hop questions. kapa.ai’s “no reranker” mistake is one fix path for buried hits; it is not a separate failure mode from ranking the wrong neighbour first.

On the generation band, three symptoms dominate once retrieval looks healthy. The model invents beyond the context: hallucination despite context. The right chunk was retrieved and buried mid-prompt: lost in the middle. Two retrieved passages disagree and the model picks badly: conflicting sources. PromptQL’s extraction and computational-reasoning failures sit on this side when the model misreads a present fact or stops short of a full verification plan. Distinguishing those bands still leaves a third band that looks like retrieval but is really how the corpus was cut and stored.

How do index and chunking failures show up in RAG?

Index and chunking failures show up as retrieval bugs when the document was never stored as a clean candidate. kapa.ai treats bad chunking as cutting a sentence, paragraph or logical idea in half so embeddings drift away from meaning, and treats poor data quality as garbage-in for the whole vector store. Barnett’s Not in Context point covers the consolidation form: the retriever finds the right document, yet truncation, weak chunking or a bad merge strategy keeps the answer out of the model’s window.

Two leaves own that band for this cluster. When the answer straddles a boundary between chunks, the symptom page is chunk boundary loss. When numbers live in rows and columns that flattening destroyed, the symptom page is why RAG gets numbers and tables wrong. Re-prompting the generator will not rejoin a split sentence or restore a lost unit column. Hierarchical or parent-document chunking is a mechanism fix that belongs with the chunking cluster after the symptom has been confirmed, not as a guess from a single bad demo answer.

Dirty ingest and missing deletes also sit here even when the chunker looks fine: a document that never entered the index cannot be retrieved, and a document that was deleted from the source but left as an orphan vector will keep answering. Those freshness forms are named with the production modes below, because teams usually notice them only after the demo corpus stops being static. With index, retrieve and generate separated, the next need is a single overview of the modes that appear once real users and a growing corpus arrive.

What are the common RAG failure modes in production?

The common RAG failure modes in production are the retrieval, generation, freshness, security and performance symptoms that a small demo corpus never exercises. Maurya’s knowledge-drift horseman is yesterday’s rate still answered confidently after the source moved. His retrieval-decay horseman is a corpus that grew from a toy set into millions of documents until the needle disappeared. kapa.ai’s embedding rot is the quiet cousin: embeddings go stale when the embedding model space improves, when roughly 10 to 15 percent of the corpus changes, or when the domain itself shifts, and teams postpone re-embedding because migrations hurt.

Freshness leaves on this map are stale index for answers that cite a version you already deleted or replaced, and retrieval quality degradation for quality that slides as the corpus or the query mix moves. Security and performance leaves are multi-tenant leakage when a missing filter returns another tenant’s documents, and rag latency when retrieval or reranking blows the budget under load. PromptQL’s context-size failures are one measured shape of latency and window pressure when multi-step retrieval piles chunks into a single prompt.

Organisational failure has its own leaf because permissions, source sprawl and undefined success criteria are not fixed by a new embedding model alone: enterprise rag failure. The evaluation-gap horseman and kapa.ai’s zero-evals mistake are how those modes stay invisible: thumbs widgets go unused, and nobody runs retrieval-level checks before blaming the generator. Naming every mode is still not a diagnosis order. The next question is which measurement isolates the stage when only “wrong answer” is in hand.

How do you find which stage of your RAG pipeline broke?

You find which stage broke by measuring one property per stage in order, instead of guessing from the final sentence. kapa.ai’s eval framework starts with a small labelled set, runs retrieval-level checks first, and only then trusts generation scores, because a failed retrieval means generation never stood a chance. Maurya’s continuous feedback loop logs queries, scores groundedness, relevancy and hallucinations with libraries such as Ragas, and regenerates synthetic tests from real traffic on a weekly or monthly cadence. Both are saying the same thing: an end-to-end vibe check cannot localise the fault.

The decision procedure that walks one measurement per stage, in order, until the failing component is isolated, lives at rag debugging. The broader method that keeps retrieval metrics, generation metrics and production checks distinct is how to evaluate a RAG system. Better chunking fixes boundary and some table failures; it does nothing for stale indexes, cross-tenant leakage, embedding drift or latency. Is it the model or the retrieval? Measure whether the correct chunk entered the top-k before you change the prompt.

When the measurements are green offline and users still complain after ship, the remaining band is organisational and operational rather than a single broken span. That is the enterprise failure question: what a clean demo hid about permissions, source sprawl and success criteria once the system left the lab.

Why do RAG pilots fail in enterprises?

RAG pilots fail in enterprises when a demo-grade pipeline meets production permissions, heterogeneous sources, undefined success criteria and a corpus that keeps moving. Barnett et al. (arXiv 2401.05856) state that validation of a RAG system is only feasible during operation, and that robustness evolves rather than being designed in at the start. Maurya’s production framing is the same gap from the other side: what works on a curated proof-of-concept set rarely survives weekly schema changes, redundant retrieval under growth, and an evaluation gap nobody funds.

The organisational depth for permissions, source sprawl and success criteria is why RAG pilots fail in enterprises. Leakage and latency often appear only at enterprise scale even when the demo looked clean, which is why those leaves sit beside the organisational one rather than inside a model-tuning checklist. A pilot that cannot say which stage failed also cannot say whether the next dollar should go to connectors, multi-tenancy, re-embedding or a smaller scope.

Production fails after a clean demo for the same structural reason the taxonomy exists: the demo never stressed freshness, scale, permissions or measurement. Stage-sort the symptom, measure the stage, then open the leaf that owns the mode. Re-chunking everything because the answer looked wrong is how pilots stall for months without changing the number that actually moved.

A RAG system returns the wrong answer in many ways, and every one of them still belongs to a stage: how the corpus was indexed and chunked, what retrieval returned, what the generator did with that window, or a cross-cutting band such as freshness, security, performance or organisational fit. Barnett et al. (2024) give seven named points and the operating lesson that robustness is earned in production. Label Studio, kapa.ai, PromptQL and Maurya’s production horsemen fill in the practitioner vocabulary around those points. The taxonomy here pins each mode to a stage and a leaf so the next change matches the fault. Measure retrieval before you blame the model, and treat a green demo as unfinished evidence until the production bands have been exercised.