Late Chunking and Contextual Retrieval

Late chunking embeds the whole document — or as much as the embedding model’s context window allows — first, then mean-pools the token vectors inside each chunk span so every chunk vector carries document context without an LLM prepend. Contextual retrieval (Anthropic, September 2024) takes the other path: an LLM writes a short situating preface (usually 50–100 tokens) and prepends it to each chunk before embedding and BM25. Both fix the lost-context problem that naive chunk-then-embed creates; they differ in where the context is attached. This page covers the mechanism, the published numbers, the cost trade, and when to pick each — against the strategy catalogue on the chunking hub and boundary methods such as semantic chunking.

What problem does lost context create in RAG?

Naive chunk-then-embed makes each chunk embedding independent: pronouns and entity references that resolve outside the chunk never reach the vector, so retrieval misses chunks that would be obvious to a human reading the full page.

Günther et al. (arXiv:2409.04701, 2024) and the Jina late-chunking write-up demonstrate this on a Berlin Wikipedia split. Against the query embedding for “Berlin,” the chunk “Its more than 3.85 million inhabitants…” scores cosine 0.708 under naive chunking versus 0.825 under late chunking; “The city is also one of the states…” scores 0.753 versus 0.850 (jina-embeddings-v2-small, their Table 1). Anthropic’s SEC-filing example is the same failure in financial prose: a chunk that only says “The company’s revenue grew by 3%” does not name the company or the quarter. Overlap windows and multi-pass heuristics can blunt this sometimes; Jina’s part I notes they carry no theoretical guarantee. Related but distinct: when the answer text itself straddles two chunks, that is chunk-boundary loss — lost context is about embeddings that cannot resolve references the chunk text leaves dangling.

How does late chunking work?

Late chunking inverts the usual order: the transformer sees the long document first; chunk boundaries are applied only when pooling token vectors into one vector per chunk.

1. Tokenize up to the model’s context length. Long-context embedders such as jina-embeddings-v2 / v3 accept on the order of 8192 tokens — roughly ten standard pages (Jina, August 2024).
2. Run the transformer once. Encoder-style embedding models use full (bidirectional) attention, so each token representation is conditioned on the other tokens in the window (Jina part II).
3. Mean-pool inside each chunk span. Boundary cues still exist — fixed token counts, sentences, or a semantic splitter — but they are used after the forward pass, which is why the method is called “late.” The resulting chunk vectors are conditional on the document, not i.i.d. like naive per-chunk embeds.

Jina’s part II ablations report that late chunking is resilient to weak boundary cues (late + fixed-token often matches or beats naive + semantic boundaries in their setup), while embedding-model strength still dominates: a weaker model with late chunking does not automatically beat a stronger model without it. The method works without extra training; optional fine-tuning on query–document–relevant-span tuples with a contrastive loss such as InfoNCE is described in the paper and part II as a further gain, not a requirement. What the embedding step can decide at all is framed on the embeddings hub.

What is contextual retrieval?

Contextual retrieval, as Anthropic published it in September 2024, prepends chunk-specific explanatory context before you embed and before you build BM25 — so both the dense and lexical indexes see situating terms the raw chunk lacked.

For each chunk, an LLM (Anthropic used Claude 3 Haiku in the post) reads the whole document plus the chunk and writes a short situating string, usually 50–100 tokens, from their published “situate this chunk” prompt. That string is prepended for Contextual Embeddings and for Contextual BM25. With prompt caching, Anthropic states a one-time cost of about $1.02 per million document tokens under their size assumptions (800-token chunks, 8k-token documents, 50-token instructions, 100-token contexts — verify live prices; as of the September 2024 post). The technique stacks with hybrid BM25+embeddings and with a reranker. For knowledge bases smaller than about 200,000 tokens (~500 pages), Anthropic also notes you can skip RAG and put the whole corpus in the prompt with caching instead.

How does late chunking differ from contextual retrieval?

Both attack lost context; they attach that context at different points in the pipeline.

• Late chunking puts context into the embedding via attention. No extra text is stored; vector count matches naive chunking; you need a long-context mean-pooling embedder; there is no LLM call per chunk at ingest.
• Contextual retrieval puts context into the indexed text as literal prepended tokens. It works with any embedder (including closed single-vector APIs); it costs an LLM call per chunk plus longer stored text; Contextual BM25 benefits because the lexical index sees the situating terms.

Jina’s part II cites a qualitative comparison in arXiv:2409.04701 where Anthropic-style enrichment performed similarly to late chunking on their setup — treat that as one paper’s observation, not a universal ranking across corpora. Neither method replaces choosing readable chunk spans for the generator. The storage extreme of keeping every token vector for late interaction is a different design owned by ColBERT and late interaction.

What published retrieval gains are reported?

Two published lines matter, and they are not interchangeable: late chunking reports BEIR nDCG@10; contextual retrieval reports a cut in top-20 retrieval failure rate (1 − recall@20). Do not subtract one paper’s percentages from the other’s.

The late-chunking paper also notes that the late-versus-naive gap grows with average document length. Prove either method on your labelled set with the protocol on chunking evaluation — published tables are starting points, not transfers.

What does late chunking cost compared with contextual retrieval?

Late chunking’s structural cost is one long-context forward pass per document (or window) at ingest, with storage identical to naive chunking — one vector per chunk. Contextual retrieval’s structural cost is an LLM call per chunk at ingest, longer indexed strings, and Anthropic’s published one-time context-generation figure under their size assumptions.

• Late chunking storage matches naive chunking. The Weaviate late-chunking blog (2024) sketches 100,000 documents × 8,000 tokens at 768-d fp32: late interaction / ColBERT-style token storage ≈ 2.46 TB (8,000 vectors per doc) versus naive or late chunking ≈ 4.9 GB (16 vectors per doc at 512-token chunks). That is a structural illustration, not a quote of your cloud bill. ColBERT depth lives on late interaction.
• Contextual retrieval pays an LLM at ingest. Anthropic’s $1.02 per million document tokens (September 2024 assumptions above) is the one-time context-generation cost with prompt caching — plus storage of original text plus 50–100 token prepends. Query path is otherwise standard retrieval over richer embeddings and BM25.
• No universal $/query for late chunking is published across providers. Measure embedding latency and price on your model and document length. Jina’s claim that late chunking is “significantly faster” than LLM enrichment is a vendor qualitative claim (part II, 2024) — useful as a direction, not a latency SLA.

Which embedding models support late chunking?

Late chunking needs three things (Jina part II checklist): token-level embeddings before pooling (single final-vector APIs usually cannot), useful long context (on the order of 8192 tokens), and mean pooling — CLS or max pooling is incompatible with span-wise mean pool.

Published demonstrations include jina-embeddings-v2-small/base-en, jina-embeddings-v3, and nomic-v1 (Jina part II). The arXiv paper states the method is generic to long-context mean-pooling embedders, not exclusive to Jina. Model choice still dominates the late-versus-naive gain. Selection criteria continue on embeddings and how to choose an embedding model.

When should you use late chunking instead of contextual retrieval?

Choose late chunking when you control a long-context mean-pooling embedder (self-host or an API that exposes token-level / late-chunk pooling), you want naive-chunk storage cost, documents are long enough that context loss shows (the paper’s uplift correlates with average document length), and you refuse per-chunk LLM spend at ingest.

• Choose contextual retrieval when your embedder is short-context or a closed single-vector API; you already pay for an LLM and prompt caching; you need BM25 to see situating terms (Contextual BM25); or a domain-specific situate prompt helps.
• Choose neither first when chunk size and overlap are still untuned — fix those knobs on chunk size and overlap, then re-measure. Both methods can still hand the generator a hard-to-read span if boundaries ignore document structure; pair with structure-aware or semantic splitters when headings or meaning boundaries matter more than context enrichment alone.

How do you implement late chunking?

The reference implementation is the jina-ai/late-chunking repository on GitHub (Günther et al.). Weaviate’s late-chunking blog notes the pooling change is small (on the order of tens of lines) and that the retrieval pipeline is unchanged — vectors ingest as ordinary chunks into Weaviate, Pinecone, Qdrant, or Milvus. Contextual retrieval starts from Anthropic’s cookbook and situate prompt; AWS documented a Bedrock Knowledge Bases custom-chunk path for the same idea (2024–2025). Runnable pinned builds live on building the pipeline; measured comparison belongs on chunking evaluation.