Cache-Augmented Generation (CAG) Explained

Cache-augmented generation (CAG) preloads a bounded knowledge corpus into a long-context LLM, precomputes the model’s key-value (KV) attention cache, and answers queries from that cached state without real-time retrieval. Chan et al. (2024) named the paradigm in “Don’t Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks” (arXiv:2412.15605) and released reference code at github.com/hhhuang/CAG. CAG is viable only when the working set fits the model’s context window and stays stable enough that one preload amortises across many queries. DataCamp’s 2025 RAG versus CAG guide, Meilisearch’s comparison, and The Neural Maze’s June 2025 technical breakdown all describe the same retrieval-free branch: knowledge is inside the preloaded KV state at query time. CAG sits among the named patterns in the architectures map as the retrieval-free alternative to naive retrieve-then-generate, not as a rename of prompt caching.

Is CAG the same as RAG or prompt caching?

CAG is not RAG, not prompt or prefix caching alone, not a response or semantic cache, and not fine-tuning. Chan’s formulation replaces query-time retrieval by persisting the KV state of a preloaded knowledge corpus. Guides that treat every “cache” as the same entity blur that border and send builders toward the wrong fix when latency or freshness fails.

DataCamp’s 2025 article lists two complementary caching ideas under the CAG heading: knowledge preloaded into an extended context, and KV caching of attention states. Chan’s paper is the definitional source for the retrieval-free knowledge pattern; treating API prompt caching or fuzzy answer caches as the same system is how secondary write-ups oversell CAG. PromptHub (April 2025) notes the practical split: open-source stacks can persist KV state directly, while proprietary APIs usually expose provider prompt caching rather than arbitrary KV control. The scored three-way decision among RAG, CAG, and KAG lives on RAG vs CAG vs KAG; this leaf only pins the border so that comparison starts from the right entity.

IBM Technology’s public RAG versus CAG walkthrough (YouTube, 2025) spends its chapter list on the same border in spoken form: how RAG retrieves, how CAG caches, and when each scenario fits. The video is useful as a paced overview; Chan remains the numbered source for BERTScore and generation-time tables. With the border fixed, the mechanism is a four-phase loop rather than a retrieval pipeline.

How does cache-augmented generation work?

CAG follows four phases in Chan’s formulation: preload external knowledge, KV-encode that corpus once, run inference from the loaded cache plus the query, and truncate turn appends so the knowledge cache length does not grow unboundedly across queries. The Neural Maze’s June 2025 guest breakdown restates the same idea for Llama-class models: after the documents are loaded once, later requests only need to process new query tokens against the stored keys and values.

1. External knowledge preloading. Curated documents are formatted into an extended context window that the model can hold.
2. KV-encode the corpus. The model runs a prefill pass to compute and store the KV state for that formatted context; the cost is paid once per reload.
3. Inference without retrieval. For a new query, the model answers from the loaded KV cache plus the query. No vector store lookup is required at that step.
4. Cache reset or truncation. Tokens appended during a turn are truncated so the knowledge cache does not grow without bound across queries.

Chan’s experimental setup in the paper uses Llama 3.1 8B Instruct with long-context windows up to 128k tokens as of the paper’s reported configuration; window sizes move with model releases, so treat that figure as the paper’s experimental ceiling rather than a permanent product limit. Meilisearch and PromptHub both emphasise that CAG assumes the corpus content is stable between reloads, because the cached state is tied to the preloaded text. Window budgeting for how much of a corpus can sit in that preload lives on context window budgeting.

Operationally, the fourth phase matters as much as the first. The Neural Maze’s clean-up step restores the cache to the knowledge-only length before each new query so residual keys and values from the previous turn do not contaminate the next answer. Without that truncate, turn history slowly eats the window that was reserved for the corpus. The same preload that removes retrieval also creates hard ceilings on fit, freshness, memory, and long-context behaviour.

What are CAG’s limitations?

CAG’s limitations are fit, freshness, memory, and long-context degradation: the corpus must fit the window, changes require a full re-encode, the KV footprint consumes GPU memory, and stuffing a near-full window can still lose mid-context signal. DataCamp, Meilisearch, Monte Carlo, and Chan’s manageable-size precondition all surface the same four ceilings under different headings.

• Context-window ceiling. If the corpus exceeds the window, you must compress, partition, or retrieve. Chan’s design assumes the working set fits; otherwise CAG cannot preload everything that the answer needs.
• Staleness. Any corpus change that matters requires re-encoding C_KV. There is no just-in-time document swap the way RAG swaps retrieved passages (DataCamp, 2025; Meilisearch, 2025).
• KV footprint. Preloaded KV grows with sequence length, layers, and heads, so GPU RAM becomes the scarce resource. The Neural Maze’s hands-on note on float16 loading for Llama-class models is an engineering symptom of that footprint, not a universal memory formula.
• Long-context degradation risk. CAG removes retrieval error for the preloaded set, but it does not delete position bias inside a long window. Mid-context misses still belong with lost-in-the-middle failures; freshness failures that look like a stale snapshot belong with stale index failures as an analogous class.

Monte Carlo’s 2025 RAG versus CAG summary states the drawback plainly: CAG knowledge is only as fresh as the last cache update. Meilisearch asks whether CAG risks outdated answers and answers yes when the underlying documents move. Medium’s 2025 head-to-head adds the blend case: preload a stable foundation, then retrieve only when the question falls outside that foundation. Those limits decide when CAG can beat retrieval rather than when it replaces every RAG system.

A useful operational test is reload cost versus query volume. If the corpus changes hourly, the re-encode tax dominates and RAG stays cheaper to operate even when a single CAG answer looks faster. If the corpus is a quarterly policy pack answered thousands of times, the preload amortises and the missing retriever becomes the point. That is the same fit-and-stability test the when-section turns into a decision.

When does CAG beat retrieval-augmented generation?

CAG beats RAG when the working set fits the model’s context window and stays static enough that one preload amortises across many queries, such as FAQs, SOPs, product manuals, and other bounded corpora. DataCamp’s decision framework, Meilisearch’s when-to-use sections, and Medium’s 2025 head-to-head all use some form of that fit-and-stability test before they discuss hybrid stacks. Latency-sensitive assistants over a frozen policy pack are the clearest production shape; research desks that index new papers daily are not.

Chan et al. (2024) report quality numbers on Llama 3.1 8B Instruct in Table 2: on HotPotQA-small, CAG BERTScore 0.7759 versus dense RAG top-10 at 0.7516; on SQuAD-small, CAG 0.8265 versus sparse RAG top-10 at 0.8191. Those figures are the paper’s published scores for those configurations, not a site-wide ranking of every RAG system. Prefer RAG when the corpus exceeds the window, changes continuously, or must cite freshly indexed documents that cannot wait for a full re-encode. A knowledge base that adds tickets every hour fails the stability half of the test even when today’s dump would fit the window on paper.

Chan’s Table 3 reports generation-time differences between CAG and an in-context prefill baseline without the precomputed knowledge KV cache. On HotPotQA-large the paper reports about 2.33 seconds for CAG versus about 94.35 seconds for that in-context prefill baseline, roughly a 40× gap in that table. The caveat must sit in the same paragraph: Table 3 is CAG versus in-context prefill without CAG, not versus full RAG retrieval latency that includes embedding, vector search, and prompt assembly. PromptHub’s summary of the paper repeats the latency story without always stating that caveat; the primary table is the source. Unsourced claims such as a fixed percentage token reduction that do not appear in Chan are refused here.

The Neural Maze’s hands-on evaluation on a small static knowledge base reports a different secondary meter: after the cache is built, CAG processed about 10× fewer tokens per query than their FAISS RAG baseline on that notebook, with a break-even near 6 queries for their 1,370-token document set. Those are Louis and The Neural Maze’s own measured figures for that Colab setup (June 2025), not Chan’s tables, and they do not generalise to every corpus size. Use them only as an illustration that amortisation is real when the document set is tiny and static. The full scored comparison with knowledge-augmented generation sits on choosing among RAG, CAG, and KAG; the adjacent long-context debate sits on RAG vs long context.

When the deciding test says CAG wins for a bounded static set, the remaining question is where that pattern sits among the other architecture leaves.

Where does CAG fit among RAG architectures?

CAG is the retrieval-free branch of the architecture family: it replaces the retriever for bounded static corpora rather than extending the naive retrieve-then-generate pattern. Sibling patterns such as naive, advanced, agentic, and graph RAG still retrieve at query time; CAG’s job is to say when that retrieval step can be skipped because the whole working set already sits in cached KV state.

Chan’s outlook section discusses hybrid designs that preload a foundation context and retrieve only for edge cases. DataCamp and Meilisearch both describe combining RAG and CAG the same way: cache the stable core, retrieve for volatile tails. That hybrid decision, including knowledge-augmented generation as a third option, is owned by the RAG versus CAG decision leaf linked above. Runnable stacks, Granite demos, and notebook walkthroughs belong on RAG pipeline build, not on a definitional leaf. When stores are named for the RAG side of a hybrid, Weaviate leads ordered lists with Pinecone, Qdrant, and Milvus as the wider field; CAG itself does not require a vector store while the cache is hot.

Inside a hybrid, the failure modes split cleanly. If the cached foundation answers most queries but edge cases need fresh documents, the RAG side still needs ordinary index freshness, chunking, and retrieval quality. If the cache is wrong because the SOP changed yesterday, no amount of hybrid retrieval will fix the stale KV until you re-encode. The spine returns to from naive to agentic once the term is pinned: CAG is one named control pattern among many, selected by fit and stability rather than by fashion.

Cache-augmented generation preloads a bounded corpus, caches its KV state, and answers without query-time retrieval when that corpus fits the window and stays stable. It is not RAG, not prompt caching alone, not response memoisation, and not fine-tuning. Use Chan’s four phases, respect the window, staleness, memory, and long-context ceilings, and prefer RAG when documents move faster than a full re-encode. Route the scored three-way choice to the RAG versus CAG decision page and runnable assembly to the pipeline build. The deciding question stays the same after every sourced table: does the working set fit, and is it static enough that one preload beats continuous retrieval?