Contextual Retrieval: Anthropic’s Chunk-Context Method

Contextual retrieval (Anthropic, September 2024) prepends a short, chunk-specific situating context — usually 50–100 tokens from an LLM — to each chunk before embedding it (Contextual Embeddings) and before building BM25 (Contextual BM25). That restores the company, quarter, or section a bare chunk left out. Under Anthropic’s published size assumptions with prompt caching, the one-time cost to generate those contexts is about $1.02 per million document tokens. The late-chunking comparison lives on late chunking and contextual retrieval; this page is the Anthropic method.

How does contextual retrieval work?

Contextual retrieval is an ingest-time transform. The query path stays ordinary retrieval — only the indexed text is richer.

1. Chunk the document as usual. Boundaries still matter; contextualization does not replace choosing size and overlap on chunk size.
2. Situate each chunk with an LLM. Anthropic’s published Claude 3 Haiku prompt places the whole document and the chunk in tags and asks for a short succinct context that situates the chunk for search — answer with the context only (Anthropic engineering post, 19 September 2024).
3. Prepend that context to the chunk. The situating text is usually 50–100 tokens.
4. Embed and index the contextualized text. Dense embeddings and BM25 both see the prepended terms.

Worked transform from Anthropic’s SEC-filing example: the original chunk “The company’s revenue grew by 3% over the previous quarter” becomes a contextualized chunk that opens with “This chunk is from an SEC filing on ACME corp’s performance in Q2 2023; the previous quarter’s revenue was $314 million…” before the same sentence. At query time you search those enriched units — you do not re-run the situate prompt per user question.

What is Contextual Embeddings versus Contextual BM25?

Anthropic names two sub-techniques that share the same prepended text:

• Contextual Embeddings embed the contextualized chunk so dense similarity can match on situating entities (company, quarter, section) that never appeared in the bare passage.
• Contextual BM25 builds the lexical index on the same contextualized string, so those situating terms are also exact-match searchable.

Their stack is embeddings alone, then embeddings plus Contextual BM25 (hybrid on contextualized text), then an optional reranker. The fusion maths for BM25 plus dense live on hybrid search; the lexical scorer itself is on BM25.

What retrieval failure-rate cuts did Anthropic measure?

Anthropic’s metric is 1 − recall@20 — the top-20-chunk retrieval failure rate — averaged across their knowledge domains with Gemini Text 004 embeddings (engineering post, 19 September 2024). Domains included codebases, fiction, ArXiv papers, and science papers.

These are failure-rate cuts on their eval, not a universal “35% accuracy” claim — that shorthand appears in secondary blogs and misstates the unit. They are also not interchangeable with Jina’s BEIR nDCG@10 late-chunking numbers; different metrics and datasets. Prove the lift on your own labelled set via chunking evaluation. Reranker choice and latency depth live on reranking.

What does contextual retrieval cost?

The expensive step is one-time, at ingest — not per query.

• Situating cost. With prompt caching (document loaded once), Anthropic published $1.02 per million document tokens assuming 800-token chunks, 8k-token documents, 50-token instructions, and 100 tokens of context per chunk (September 2024).
• Larger indexed text. Every stored chunk carries the original plus the prepend, so embedding tokens and BM25 document length rise.
• Query path. Situating does not add an LLM call per user question. Hybrid search and reranking add their own query-time cost and latency — Anthropic notes the rerank trade-off explicitly.

When should you use contextual retrieval?

Use contextual retrieval when bare chunks lose the entities that make them findable — and skip the whole RAG stack when the corpus is small enough to prompt whole.

• Strong fit: long docs with pronouns, cross-section references, or time-bound metrics (Anthropic’s SEC example); knowledge bases too large for a single prompt; pipelines that can afford a one-time situate pass and a BM25 index on contextualized text.
• Skip RAG first: if the knowledge base is smaller than about 200,000 tokens (~500 pages), Anthropic recommends putting the whole corpus in the prompt with caching instead of retrieval.
• Prefer late chunking instead when you control a long-context mean-pooling embedder and refuse per-chunk LLM spend — decision depth on late chunking vs contextual retrieval.

Anthropic’s implementation notes still apply: tune chunk boundaries, consider domain-specific situate prompts, and always run evals. In their tests, passing the top 20 chunks beat top 5 or 10 — still experiment on your use case.

How does contextual retrieval differ from late chunking?

Both fix lost context; they attach that context in different places. Contextual retrieval writes literal tokens and prepends them — it works with any embedder and also lifts BM25, at the price of an LLM call per chunk at ingest. Late chunking runs the embedding model over a long window first, then mean-pools inside each chunk span — no prepended text, no situate LLM, but you need a long-context mean-pooling embedder. Full comparison, BEIR nDCG figures, and the choose-when frame live on late chunking and contextual retrieval. Neither method replaces readable chunk spans for the generator.

How do you implement contextual retrieval?

The pattern is: situate each chunk with the published prompt → prepend → embed the contextualized text → build BM25 on the same text → optionally rerank. Anthropic’s Claude cookbook, DataCamp’s LangChain tutorial, and Together.ai’s hybrid demo all follow that shape. Contextualized vectors land in stores such as Weaviate, Qdrant, Pinecone, or Milvus, with a sparse/BM25 index beside them for Contextual BM25. For a minimal runnable pipeline see building the pipeline; for hybrid fusion see hybrid search; for the rerank stage that produced Anthropic’s 67% failure-rate cut see reranking; for the retrieval cluster see retrieval.