When You Should Not Use RAG
The corpora and question types where retrieval adds cost and no accuracy — stated plainly.
The short answer
Skip RAG when retrieval would not change the answer. If the corpus fits without an index, the gap is behaviour or prompting, the truth is a live lookup or tool call, or the task is structured or creative rather than open factual search over a large changing corpus — do not stand up retrieval. Reach for RAG only when a large, changing corpus needs selective factual lookup with citations or access control. No invented failure rates. (As of July 2026.)
When should you not use RAG?
You should not use RAG when adding a retrieval step would not change the answer — or would make it worse. Retrieval-augmented generation fetches passages from an external corpus and conditions generation on them. That helps only when the model needs a selected slice of knowledge that is too large or too changeable to stuff into the prompt, and the access pattern is factual lookup. If those conditions fail, the index, embedder, and retriever are cost and failure modes without a payoff. The comparisons that decide the positive cases live on RAG vs long context and RAG vs fine-tuning; this page is the honest negative. For the full decision set, return to RAG decisions.
Which situations make RAG the wrong tool?
When the deciding test is ambiguous, score the situation against the common skip cases. This is a directional guide, not a benchmark — each row names which alternative the situation favours, and why. No failure percentages are invented here; where a peer states a threshold, treat it as their claim, not ours.
| Situation | Skip RAG? | Use instead | Why |
|---|---|---|---|
| Corpus fits the window and stays mostly static | Yes | Long context / fat system prompt | An index adds sync and retrieval failures when stuffing already gives full recall. |
| Gap is format, tone, or rubric — not missing documents | Yes | Fine-tuning (after prompting) | RAG changes what the model knows at runtime; it does not bake consistent behaviour into weights. |
| Task is under-specified instructions | Yes | Prompt engineering | Cheaper than either retrieval or training; reveals whether a real knowledge gap exists. |
| Answer is live state or a known entity ID | Yes | Tools / SQL / API | Documents describe the world; they do not authoritatively represent balances, inventory, or flags. |
| Classify, extract, or route into a schema | Yes | One LLM call + typed schema | The input is already in the request; retrieval is not the bottleneck. |
| Answer requires synthesis across many sources | Often | Full-context pass or structured extract | Vector top-k returns fragments; emergent patterns across a corpus are not a single chunk. |
| Creative writing or well-known general facts | Yes | Plain LLM | Grounding external passages does not improve originality or textbook knowledge the model already has. |
| You cannot enforce auth at retrieval time | Yes — fix first | Permissioned fetch after auth | A shared index without ACLs is a leak path; prompt wording is not a security boundary — see tenant leakage. |
| Ultra-tight latency on tiny knowledge | Often | Cached prompt or plain call | Embed → search → optional rerank → generate adds hops before the model ever runs. |
Read the first row. If the working set already fits and barely moves, the rest of the table rarely overturns a skip. LindleyLabs (2026) frames the break-even the other way: RAG earns its keep when the corpus is large, updates frequently, and the primary access pattern is factual lookup — miss any of those three and you are paying the retrieval tax for the wrong problem.
When is long context enough without RAG?
Long context — or stuffing a fat system prompt — is enough when the working set fits the model’s context window and stays static enough that an index would add cost without accuracy. A company handbook, a small FAQ, or a frozen deal-room pack that you re-query without the files changing is the classic case: perfect recall, no chunker, no retriever, no stale vectors. Modern windows make that path wider than it was in 2023; the size, cost, and “is RAG dead?” comparison — including sourced LongBench margins from Li et al. (2024) — live on RAG vs long context. Prefer “does it fit your window?” over any universal token cutoff published as marketing.
When should you use fine-tuning or prompting instead of RAG?
Use fine-tuning when the gap is behaviour — a format, tone, or rubric the model should always apply — and use prompting first when the instructions are simply under-specified. RAG injects documents at query time; it does not teach the model to emit your JSON schema or hold your brand voice without re-explaining it every call. Many teams reach for a vector database when five well-chosen examples in the prompt would have closed the gap. Exhaust prompting, then choose RAG vs fine-tuning by whether the remaining gap is knowledge or behaviour; the cheaper try-first path is RAG vs prompt engineering.
When should you use tools or SQL instead of RAG?
Use tools, APIs, or SQL when correctness depends on live system state or a deterministic lookup —
not on searching prose. Order #48291, the current inventory count, a feature flag, or
“can this user access X right now?” are authoritative in your services; a PDF indexed last Tuesday is not.
If the ticket thread or form fields are already in the request, assemble that context into the prompt —
that is context assembly, not retrieval. Classification, extraction, and routing are the same family:
map input → schema with one LLM call. When you do retrieve documents, treat every chunk as
untrusted input — retrieved text expands the
prompt-injection surface rather than eliminating it. The broader
alternatives map is on what are the alternatives to RAG.
When should you still use RAG?
Still use RAG when the corpus is too large to stuff, changes often enough that fine-tunes go stale, and users need factual lookup with citations or per-user access control. Support over thousands of changing product docs, policy libraries that update with regulation, and multi-tenant document sets are the canonical fits — see what RAG is for the mechanism. Before you buy an embedding pipeline, prove the need: define golden questions from real workflows, try structured or full-text retrieval first, and add semantic search only when those simpler paths miss questions that block the feature. 475 Cumulus (June 2026) publishes a similar prove-before-buy sequence for engineering leads; the evaluation spine on this site is how to evaluate a RAG system. Shipping RAG without that gate creates confidence without reliability.
What should you use instead of RAG?
When the skip test fires, pick the lighter tool that matches the gap — this page decides whether to skip; the siblings own depth:
- Long context / stuffing — bounded static packs that fit the window → RAG vs long context.
- Fine-tuning — fixed behaviour, format, or tone → RAG vs fine-tuning.
- Prompt engineering — under-specified instructions → RAG vs prompt engineering.
- Tools / SQL / full-text + filters — live state, entity IDs, or structured search before vectors → alternatives to RAG.
- Refuse out-of-scope — “chat over all our data” with no workflow often needs a narrower product, not a bigger index.
When should you not use RAG?
Skip RAG when adding retrieval would not change the answer. If the corpus fits the context window and stays mostly static, stuff or cache it. If the gap is behaviour or under-specified instructions, use fine-tuning or prompting. If the truth is live state or a known entity ID, call tools or SQL. Reach for RAG only when a large, changing corpus needs selective factual lookup with citations or access control.
Is RAG always overkill for a small FAQ?
Usually yes, if the FAQ fits your model's context window and barely changes. A fat system prompt or long-context pack gives full recall without an embedder, index, or retrieval failure mode. Revisit retrieval only when the set outgrows the window, updates often, or needs per-user document filtering — the size and cost comparison is on RAG vs long context.
Can I skip a vector database?
Yes, whenever RAG itself is the wrong tool — and often even when you need light search. Start with context already in the request, SQL or API lookups, and full-text search with metadata filters. Add a vector index only when evals show a semantic gap that blocks the feature. Skipping the vector database is correct whenever retrieval would not change the answer.
Does 'RAG is dead' mean you should skip retrieval?
No. 'RAG is dead' usually argues that huge context windows replace retrieval. Larger windows change what fits in one prompt; they do not erase proprietary corpora, freshness, citations, or per-user access. When stuffing truly fits and stays static, skip RAG for that reason — not because of the slogan. The sourced comparison is on RAG vs long context.
What should I try before building RAG?
In order: clarify the prompt with examples; stuff or cache anything that fits and stays static; use tools or SQL for live or deterministic lookups; try full-text plus filters on a golden set. Add embeddings only when those paths miss questions that block the feature. Prove the need with evaluation before standing up an index — see how to evaluate a RAG system.