RAG vs Long Context: Is RAG Dead?

Is RAG dead now that context windows are huge?

No — million-token windows did not end retrieval. A larger context window lets a model read a bigger prompt in one pass; it does not give the model your private corpus, keep that corpus current, or name which file an answer came from. Douwe Kiela (Contextual AI, April 2025) restates the original RAG problems — proprietary data, knowledge cutoffs, and attribution — and notes they still hold after every context-window launch. Unstructured (Maria Khalusova, October 2024) makes the scale point concrete: a 2-million-token window is on the order of a few thousand pages of English text — roughly enough for about ten typical 300–400-page annual reports, not an enterprise knowledge base measured in terabytes.

Benchmarks do not say “always RAG” either. Li et al. (2024, arXiv:2407.16833) find that when resources are sufficient, long-context prompting consistently beats RAG on LongBench / ∞Bench averages — by about 7.6 percentage points for Gemini-1.5-Pro, 13.1 for GPT-4O, and 3.6 for GPT-3.5-Turbo — while RAG remains relevant because it cuts input tokens (and therefore API cost). Their analysis also finds RAG and long-context predictions identical on more than 60% of queries, so the expensive full-window path is often redundant. When retrieval would add cost without accuracy, that is a different decision — see when you should not use RAG. For the full set of architecture choices, return to RAG decisions.

When should you use RAG vs long context?

Every multi-factor comparison buries the decision under four or five axes. It usually reduces to one: does the working set fit the context window, and stay static enough that stuffing it is cheaper than retrieving?

• Fits and stays static → long context. A bounded pack — one deal room, one report set, a frozen codebase snapshot — that you will re-query without the underlying files changing. Optional prompt caching can amortise the first full read; budgeting tokens inside that window is context-window budgeting.
• Larger than the window, changes often, or needs citations / per-user ACL → RAG. Retrieve the relevant subset at query time, update the index when sources move, and filter by permission before anything hits the model.

Ask that first. Most of the time it answers before you reach the table. If the gap is not knowledge at all but a fixed format, tone, or skill, that is the sibling decision RAG vs fine-tuning. Ops cost at corpus scale is modelled on what a RAG system costs to run.

What’s the difference between RAG and a long context window?

When the deciding question lands in the middle, score both on the factors that matter. This is a directional comparison, not a benchmark — the ratings say which approach the factor favours, and why. No invented latency or dollar races: where a public paper or list price exists, it is named; elsewhere the cell stays qualitative.

Read the first row. If your corpus does not fit, the rest of the table rarely overturns it. Stuffing a long prompt also grows lost-in-the-middle risk: Liu et al. (2023; TACL 2024) measured U-shaped position bias — models use edges better than the middle — so a bigger window is not free accuracy. For list-price scale, Redis (February 2026) cites OpenAI GPT-4.1 rates of $2.00 per million input tokens and $8.00 per million output tokens as of that post; verify current list prices before you budget — the point is that every token in the window is billable whether the model needs it or not.

When should you use both RAG and long context?

The binary framing is a trap: production systems often use both, because cheap retrieval and expensive full-window reasoning are independent levers. Retrieve to filter the corpus, then let a long-context model reason over the retrieved set (or over full documents linked from a retrieved summary). Li et al. (2024) call one such pattern Self-Route: the model first answers from retrieved chunks and reflects on whether the evidence is enough; only “unanswerable” queries escalate to the full long context. On their suites, Self-Route cut computation cost by about 65% for Gemini-1.5-Pro and 39% for GPT-4O versus always using long context, while staying comparable to the long-context baseline — possible because RAG and LC predictions matched exactly on 63% of queries. The retrieval spine is still a RAG pipeline; the long window is reserved for the queries that need it.

What are the use cases for RAG and long context?

Where the deciding question is obvious, so is the tool:

• Reach for RAG: internal knowledge assistants over a corpus that grows past any window; support or policy answers that must stay current; answers that must cite a source; multi-tenant systems that filter documents by user permission before generation.
• Reach for long context: summarising or analysing a bounded, mostly static pack — a fixed report set, a single matter file, a snapshot of a repository — when your latency and token budget can absorb a full-window pass and you do not need per-document ACLs inside that pack.

If retrieval would only add cost, see when you should not use RAG. If the gap is behaviour (format, tone, rubric) rather than documents, see RAG vs fine-tuning.