Contextual Retrieval: Anthropic’s Chunk-Context Method

Contextual retrieval (Anthropic, 19 September 2024) prepends a short, chunk-specific situating context, usually 50 to 100 tokens from an LLM, to each chunk before embedding it (Contextual Embeddings) and before building BM25 (Contextual BM25). That restores the company, quarter, or section a bare chunk left out when documents were split. Under Anthropic’s published size assumptions with prompt caching, the one-time cost to generate those contexts is about $1.02 per million document tokens. The late-chunking comparison lives on late chunking as an alternate attachment; the parent map of retrieval choices is the retrieval hub.

How does contextual retrieval work?

Contextual retrieval is an ingest-time transform. The query path stays ordinary retrieval; only the indexed text is richer. Anthropic’s engineering post and the Claude contextual-embeddings cookbook both walk the same four steps: chunk the document as usual, ask an LLM to situate each chunk inside the whole document, prepend that short context, then embed and index the contextualized string for dense search and for BM25.

Anthropic’s published Claude 3 Haiku situate prompt places the whole document and the chunk in tags and asks for a short succinct context that situates the chunk for search, answering with the context only. Chunk boundaries still matter: contextualization does not replace choosing size and overlap on chunk size and overlap. Box’s September framing of the same method emphasises that the stored text, not the query, carries the missing entity and time labels.

Worked transform from Anthropic’s SEC-filing example: the original chunk “The company’s revenue grew by 3% over the previous quarter” becomes a contextualized chunk that opens with “This chunk is from an SEC filing on ACME corp’s performance in Q2 2023; the previous quarter’s revenue was $314 million…” before the same sentence. At query time you search those enriched units; you do not re-run the situate prompt per user question. Together.ai’s Contextual RAG guide and the Claude cookbook both keep that ingest-only situate boundary when they wire hybrid retrieval and optional rerank.

Traditional RAG fails the ACME question because the gold span never names the company or the quarter. Anthropic’s primer walks the standard hybrid path (chunk, embed, BM25, fuse, prompt) and then shows why that path still destroys context: splitting a long SEC filing into a few hundred tokens per chunk strips the identifiers the query actually uses. Contextual retrieval does not change the query encoder; it changes what each stored unit means before either channel scores it. That is why the situate call reads the whole document once per chunk at ingest and why prompt caching matters: the document prompt is reused across chunks instead of resent for every call. The situate prompt Anthropic published is deliberately short: it asks only for context that improves search retrieval of the chunk, and it forbids any answer beyond that succinct context. That constraint keeps the prepend near 50 to 100 tokens so the indexed unit stays searchable rather than becoming a second summary document. Custom prompts can add a domain glossary when key terms are defined only elsewhere in the corpus, which Anthropic lists as an implementation consideration alongside chunk-boundary tuning.

What is Contextual Embeddings versus Contextual BM25?

Anthropic names two sub-techniques that share the same prepended text. Contextual Embeddings embed the contextualized chunk so dense similarity can match on situating entities (company, quarter, section) that never appeared in the bare passage. Contextual BM25 builds the lexical index on the same contextualized string, so those situating terms are also exact-match searchable.

Their published stack is embeddings alone, then embeddings plus Contextual BM25 (hybrid on contextualized text), then an optional reranker. Box restates the same dual channel: the vector moves closer to entity-bearing queries, and BM25 can hit “ACME Q2 2023 revenue” because those tokens now live on the chunk. The fusion maths for BM25 plus dense live on combining BM25 with vectors; the lexical scorer itself is on BM25 keyword search.

Anthropic also names weaker priors they evaluated: generic document summaries added little, and hypothetical document embedding plus summary-based indexing underperformed. Those alternate query-time expansions belong on HyDE, not as substitutes for the ingest prepend.

Practically, Contextual BM25 is the reason situating helps keyword search at all. Without the prepend, BM25 cannot match “ACME” or “Q2 2023” against a chunk that only says “the company” and “the previous quarter.” With the prepend, those tokens enter the lexical postings list for that chunk. Contextual Embeddings solve the same gap for dense search when the query and the bare chunk share meaning but not surface form. Running both channels on the same enriched string is what Anthropic means by stacking the two sub-techniques before any reranker runs.

What retrieval failure-rate cuts did Anthropic measure?

Anthropic’s metric is 1 − recall@20, the top-20-chunk retrieval failure rate, averaged across their knowledge domains with Gemini Text 004 embeddings (engineering post, 19 September 2024). Domains included codebases, fiction, ArXiv papers, and science papers. The baseline failure rate on that top embedding configuration was 5.7%.

These are failure-rate cuts on their eval, not a universal “35% accuracy” claim. Secondary blogs sometimes compress the first step into that shorthand and misstate the unit. The numbers are also not interchangeable with Jina’s BEIR nDCG@10 late-chunking figures: different metrics and datasets. Prove the lift on your own labelled set via chunking evaluation.

The Claude cookbook repeats the Contextual Embeddings 35% failure-rate cut as the dense-only step, and Box cites the 67% stacked cut when a reranker sits on top. Reranker choice and latency depth live on cross-encoder and listwise reranking. Anthropic notes Voyage embeddings were strong alongside Gemini in their bake-off; that is their embedding ranking, not a site-wide winner declaration.

Read the table as a stack, not as interchangeable marketing percentages. The 35% cut is Contextual Embeddings alone against their baseline. The 49% cut adds Contextual BM25 on the same contextualized text. The 67% cut adds a Cohere reranker that scores an initial top-150 candidate set down to top-20. Dropping any stage means you are no longer claiming Anthropic’s published stacked result. Domains in their average included codebases, fiction, ArXiv, and science papers; your corpus may move each absolute failure rate even when the relative ordering of stages stays useful. Anthropic’s appendix also compares retrieval at top-5, top-10, and top-20; their headline graphs use top-20 because that depth was the most performant of the three options they tried. More chunks raise the chance the gold span is present and also raise distraction risk for the generator, so the right depth is still an eval question on your set rather than a copied constant.

What does contextual retrieval cost?

The expensive step is one-time, at ingest, not per query. With prompt caching (document loaded once), Anthropic published $1.02 per million document tokens assuming 800-token chunks, 8k-token documents, 50-token instructions, and 100 tokens of context per chunk (September 2024). The Claude cookbook restates the same cost example and points operators at cache-hit logs when they run the situate loop.

Every stored chunk also carries the original plus the prepend, so embedding tokens and BM25 document length rise. Situating does not add an LLM call per user question. Hybrid search and reranking add their own query-time cost and latency; Anthropic notes the rerank trade-off explicitly when the initial candidate set is large (their tests used top 150 before cutting to top 20).

Anthropic’s prompt-caching feature claims for longer prompts (latency cut greater than 2×, costs cut up to 90%) are about caching the whole knowledge base when it fits, not about the situate loop itself. Those figures matter when you are deciding whether to skip RAG entirely.

Two other cost surfaces are easy to miss. First, indexed text length grows by the prepend, so vector storage and BM25 index size both rise roughly with the added 50 to 100 tokens per chunk. Second, regenerating contexts after a large corpus rewrite is another one-time bill; incremental document updates need a policy for which chunks are resituated. Neither cost appears in the $1.02/M headline, which assumes a clean first pass under Anthropic’s size assumptions.

When should you use contextual retrieval?

Use contextual retrieval when bare chunks lose the entities that make them findable, and skip the whole RAG stack when the corpus is small enough to prompt whole. Anthropic’s strong-fit case is long documents with pronouns, cross-section references, or time-bound metrics (the SEC ACME example). Pipelines that can afford a one-time situate pass and a BM25 index on contextualized text are the other fit condition.

If the knowledge base is smaller than about 200,000 tokens (about 500 pages), Anthropic recommends putting the whole corpus in the prompt with caching instead of retrieval. Prefer late chunking when you control a long-context mean-pooling embedder and refuse per-chunk LLM spend; that decision depth sits on choosing late chunking or contextual retrieval.

Anthropic’s implementation notes still apply: tune chunk boundaries first, consider domain-specific situate prompts (for example a glossary defined only in other documents), and always run evals. In their tests, passing the top 20 chunks beat top 5 or 10; still experiment on your use case. Response generation can improve when the prompt distinguishes situating context from the original chunk text.

Skip contextual retrieval when the failure is not lost entity context. Vocabulary mismatch between user words and document words is a different failure class on vocabulary mismatch. Wrong chunk spans that split an answer across boundaries need chunking fixes before an LLM prepend can help. If first-stage recall is already high and the generator ignores the gold span, the next meter is generation and lost-in-the-middle behaviour, not situating. The 200,000-token skip-RAG threshold is Anthropic’s published rule of thumb for when a longer prompt with caching can replace retrieval entirely. Above that size, contextual retrieval is the scalable path they propose once ordinary chunk-and-embed RAG starts losing entity context. Below that size, paying for situating every chunk is usually the wrong spend compared with caching the corpus in the prompt.

How does contextual retrieval differ from late chunking?

Both fix lost context; they attach that context in different places. Contextual retrieval writes literal tokens and prepends them. It works with any embedder and also lifts BM25, at the price of an LLM call per chunk at ingest. Late chunking runs the embedding model over a long window first, then mean-pools inside each chunk span: no prepended text, no situate LLM, but you need a long-context mean-pooling embedder.

Full comparison, BEIR nDCG figures, and the choose-when frame live on the late-chunking mechanism page. Neither method replaces readable chunk spans for the generator, and neither is a substitute for fixing broken boundaries on chunk size before you pay for situating.

Anthropic’s post never names late chunking; the bridge exists because both techniques answer the same lost-context failure class with different attachment points and cost shapes. Teams that already standardised on a long-context embedder with mean pooling often prefer late chunking to avoid per-chunk LLM spend. Teams that need BM25 to see situating terms, or that cannot change the embedder, stay on the Anthropic prepend. Measuring both on the same labelled set is the only honest choose-when rule.

How do you implement contextual retrieval?

The pattern is: situate each chunk with the published prompt, prepend, embed the contextualized text, build BM25 on the same text, optionally rerank. Anthropic’s Claude cookbook, Together.ai’s Contextual RAG guide, and public GitHub ports of the Anthropic recipe all follow that shape. Contextualized vectors land in stores such as Weaviate, Qdrant, Pinecone, or Milvus, with a sparse or BM25 index beside them for Contextual BM25.

Together.ai’s guide emphasises hybrid fusion after contextualization and shows how RRF-style merging can sit on top of the enriched channels; fusion depth still belongs on hybrid retrieval after contextualization. For a minimal runnable pipeline see building the pipeline; for the rerank stage that produced Anthropic’s 67% failure-rate cut see the reranking stage; for the retrieval cluster see retrieval methods in RAG.

Pin the situate model and the embedding model, log cache hits on the document prompt, and measure 1 − recall@k on a labelled set before and after contextualization so the published 35% / 49% / 67% cuts are treated as Anthropic’s result, not as your production SLA.

A minimal production checklist looks like this. Keep the situate prompt versioned beside the chunker version. Write the prepended context into a dedicated field so generation prompts can label context versus source span. Build Contextual BM25 on the same string you embed. Start without a reranker, measure failure rate, then add a top-150 to top-20 rerank only if the meter moves enough to justify query latency. GitHub ports of Anthropic’s recipe and Together.ai’s guide are useful for wiring; they are not substitutes for your labelled eval. Store the original chunk id and parent document id beside the contextualized text so you can audit which situate version produced a hit. When documents change, resituate only the affected parents rather than the whole corpus unless the situate prompt itself changed. That incremental policy keeps the $1.02/M-class cost from becoming a recurring full reindex tax.

Box’s product framing adds one operational reminder that fits the same stack: reranking is especially useful after contextualization because candidates already carry entity and time labels, so a cross-encoder can judge answerability rather than guessing which ambiguous “the company” span was meant. That still does not move reranker model choice off the reranking hub; it only explains why Anthropic’s largest published cut appears on the stacked configuration.

Contextual retrieval prepends LLM-written situating text so dense search and BM25 both see the entities bare chunks lost at split time. Anthropic’s September 2024 eval cut top-20 failure from 5.7% to 3.7%, 2.9%, and 1.9% across Contextual Embeddings, plus Contextual BM25, plus Cohere rerank. Pay the one-time situate cost when the corpus is large and ambiguous; prompt the whole knowledge base under about 200,000 tokens; route late-chunking attachment and BEIR comparisons to the late-chunking sibling.