Why RAG Systems Fail: The Complete Failure Taxonomy
Every way a RAG system returns the wrong answer, sorted by the stage that caused it — with the symptom that identifies each one.
A RAG system fails in many ways, but every failure happens at a specific stage of the pipeline — and the stage that caused it is what tells you how to fix it. This taxonomy is sorted that way: not alphabetically, not by frequency, but by where in the pipeline the fault lives. Find your symptom, land on the stage, go to the page that fixes it.
Why do RAG systems fail?
Almost always in the wrong place from where people look. Most RAG failures happen before the model ever generates a word — in retrieval and chunking, not in the LLM. The model is handed the wrong context and answers it faithfully; blaming the model is the common misdiagnosis. So failures group into three pipeline zones plus a set that cuts across all of them:
- Index & chunk — the answer was mis-stored: split across a boundary, or a document that never became a retrievable candidate.
- Retrieve — the wrong passage came back, or the right one never did. This is where the largest share of failures live.
- Generate — retrieval was fine and the answer is still wrong: invented, or the right context was ignored.
- Cross-cutting — freshness, security and performance faults that aren’t tied to a single stage but bring the whole system down.
The map pins every failure to its zone, so you diagnose by where, not by guessing.
Why is your RAG not working? Find the symptom
Match what you are seeing to a symptom below — described in the words you’d use, not the jargon — and follow it to the failure that explains it and the fix that matches the cause.
Retrieval — the right chunk isn’t what comes back
Generation — retrieval was fine, the answer isn’t
Freshness — the index drifts out of date
Security & performance
Don’t know your cause yet?
Start with detection. Rather than guess which stage broke, run one measurement per stage and let the numbers point. The RAG debugging procedure →
Why does my RAG return wrong answers?
Usually because retrieval fetched the wrong passage, not because the model reasoned badly. The model answers faithfully from whatever context it is given, so a wrong retrieved chunk produces a confident wrong answer. Check the retrieved chunks first: if the correct one isn't in the top-k, it is a retrieval failure — start at the wrong-chunk failure.
Why do RAG systems fail in production but work in the demo?
Because the failures that bite in production are the cross-cutting ones a demo never exercises: the index goes stale as documents change, retrieval quality drifts as the corpus grows, permission filters leak across tenants, and latency blows its budget under load. A clean demo corpus hides all four. The enterprise-failure page covers these organisational and scale faults.
Is it the model or the retrieval?
Retrieval, far more often than people assume — most RAG failures happen before the LLM sees the query. Do not guess: measure one thing per stage in order. If the correct chunk is absent from the retrieved set, the fault is upstream of the model; if it was retrieved and still ignored, the fault is in generation. The debugging procedure isolates it in under an hour.
Does better chunking fix RAG failures?
It fixes some, not all. Chunking changes fix boundary-loss and table failures directly, and improve retrieval precision. But they do nothing for stale indexes, cross-tenant leakage, embedding drift or latency — those live outside the chunker. Diagnose the stage first, then reach for the fix that matches it rather than re-chunking and hoping.