Cache-Augmented Generation (CAG) Explained

Cache-Augmented Generation (CAG) preloads a bounded knowledge corpus into a long-context LLM and precomputes the model’s key-value (KV) attention cache so queries can be answered from that cached state without real-time retrieval. CAG works only when the corpus fits the window and stays stable enough to amortise the one-time preload.

Is CAG the same as RAG or prompt caching?

CAG is routinely confused with other “cache” patterns, but CAG’s border is clear: it replaces retrieval at query time by persisting the KV state of a preloaded corpus.

How does cache-augmented generation work?

CAG follows the same four phases in the Chan formulation: preload external knowledge; encode the KV cache; run inference from that cached state; and keep the cache bounded across turns.

1. External knowledge preloading. Curated documents are formatted into an extended context window.
2. KV-encode the corpus. The model runs a prefill pass to compute and store the KV state for that formatted context.
3. Inference without retrieval. For a new query, the model answers from the loaded KV cache plus the query—no retriever step is required.
4. Cache reset / truncation. The turn appends must be truncated so the cached knowledge does not grow without bound across queries.

Because the cache is tied to the preloaded corpus, CAG assumes the corpus content is stable between reloads.

What are CAG’s limitations?

CAG’s limitations are mostly about fit and freshness: the corpus must fit, it must not change too often, and the KV cache consumes compute and memory.

• Context-window ceiling. If the corpus exceeds the window, you must compress or partition it; otherwise you cannot preload everything.
• Staleness. Corpus changes require re-encoding the KV cache; you do not get just-in-time document swaps.
• KV footprint. The cached KV grows with sequence length and model structure, so GPU memory becomes the practical bottleneck.
• Long-context degradation risk. Preloading removes retrieval error but does not erase long-context issues such as loss of mid-context signal.

When does CAG beat retrieval-augmented generation?

CAG beats RAG when the working set fits the model’s context window and stays stable enough that one preload amortises across many queries. A key practical test is whether “fit and stability” is true for your corpus; if it is, CAG can avoid the query-time retrieval step.

In Chan et al. (2024) with Llama 3.1 8B, CAG reports higher BERTScore than dense RAG on HotPotQA-small (0.7759 vs 0.7516) and on SQuAD-small (0.8265 vs 0.8191) as shown in their Table 2. Their Table 3 also reports generation-time differences between CAG and an in-context prefill baseline (without precomputed KV cache), and their reported comparison includes the important caveat that the Table 3 generation-time numbers are not a direct “CAG vs full retrieval-latency RAG” benchmark.

When your corpus is large, frequently changing, or does not stay within the window budget, RAG remains the safer default because it can retrieve fresh documents instead of rebuilding cached state.

Where does CAG fit among RAG architectures?

CAG is a retrieval-free branch in the architectures family: it replaces the retriever for bounded static corpora rather than extending the naive retrieve-then-generate pattern. For the broader trade-off question “when to choose which,” route to the scored comparison on RAG vs CAG vs KAG, and for the general retrieval pipeline and cost control route to RAG pipeline build and context window budgeting.