Contextual Compression: Shrinking Retrieved Text Before Generation
Extractive and abstractive compression, the recall you keep, and the token cost you avoid.
Contextual compression in RAG is a post-retrieval step that shrinks each retrieved chunk to the spans that answer the query before those tokens enter the generator prompt — cutting noise and cost without changing the retriever. It is the compress stage inside context assembly, after budget and (usually) after deduplication.
How does extractive compression differ from abstractive compression?
Extractive compression keeps original sentences or spans scored as relevant to the query; abstractive compression rewrites or summarizes with an LLM into fewer new tokens. Extractive is fast and keeps the source wording — and can still miss nuance if the scorer drops a needed sentence (TheCodeForge’s production notes; the EXIT extractive line; theeazyware’s sentence-reranker pattern). Abstractive reaches higher compression ratios — TheCodeForge cites up to about 80% on survivors as an upper-bound anecdote, not a site law — but adds latency and cost per compression call and can invent phrasing, so keep citations tied to source chunk ids. A common production hybrid runs extractive first, then abstractive only on what remains. Prompt-token pruning (LLMLingua / LongLLMLingua) is a third family: it drops tokens by estimated importance rather than rewriting prose.
How does a contextual compression pipeline work?
A contextual compression pipeline retrieves (or retrieves then reranks), runs a query-conditioned compressor on each candidate or on the set, concatenates the compressed docs under the evidence budget, then generates. LangChain (April 2023) introduced DocumentCompressor.compress_documents(documents, query) and ContextualCompressionRetriever so the base retriever stays unchanged while post-processing drops irrelevant text. Typical compressors: a small-model or chain extractor; an embeddings filter with a similarity threshold; a sentence-level cross-encoder keep-top-n; or LLMLingua-style token prune. Compress after you know rank — not before. Respect the evidence cap on context-window budgeting after compression, and size the candidate set with top-k.
When does contextual compression help or hurt?
Contextual compression helps when retrieved chunks are mostly irrelevant filler around a small relevant span. LangChain’s problem statement is blunt: irrelevant text distracts the model and steals precious prompt space; theeazyware notes a chunk can be roughly 90% off-topic for a narrow question. Shorter, denser context also reduces the mid-prompt mass that feeds lost in the middle — context ordering still places what you keep. Compression hurts when the compressor drops rare must-keep facts — policy exceptions, drug interactions, exact numbers. Atlan (updated May 2026) states unmanaged compression can remove details that make answers correct and traceable; TheCodeForge reports over-aggressive semantic clustering losing a critical warning in a healthcare chatbot. Compress when signal is sparse inside long chunks; skip or raise the keep-threshold when every sentence is load-bearing, or when evaluation shows the answer span missing after compress.
What does contextual compression cost?
Contextual compression costs an extra model or scorer call per query (or per chunk) and pays back only when generator input tokens saved × generator price (plus latency) exceed that cost. Microsoft’s LongLLMLingua project page reports published anchors — not guarantees for your stack: on NaturalQuestions, up to 21.4% performance boost with about 4× fewer tokens on GPT-3.5-Turbo; 94.0% cost reduction on LooGLE; end-to-end latency 1.4×–2.6× faster when compressing roughly 10k-token prompts at 2×–6× (Jiang et al. / MS Research). Theeazyware’s rule of thumb is that the tradeoff often pays above about 2K generator input tokens — verify on your prices. Break-even checklist: measure mean tokens before and after compress; price the compressor against the generator; measure answer faithfulness on a held-out set. Never ship compress-only on cost without quality.
How do you implement contextual compression?
You implement contextual compression by wrapping your retriever with a query-conditioned compressor — LangChain’s ContextualCompressionRetriever or an equivalent in your stack — starting extractive or embeddings-filter, and escalating to abstractive or LongLLMLingua only if break-even and evaluation pass. Pin library versions and show outputs in building the pipeline; confirm gold spans still survive compression before production.
What is contextual compression in RAG?
Contextual compression is a post-retrieval step that shrinks each retrieved chunk to the spans that answer the query before those tokens enter the generator prompt. The retriever stays the same; the packing stage drops noise so cost and distraction fall.
How does extractive compression differ from abstractive?
Extractive keeps original sentences or spans scored relevant to the query. Abstractive rewrites or summarizes with an LLM into fewer new tokens — higher compression, extra call cost, and a risk of invented phrasing. A common hybrid runs extractive first, then abstractive only on survivors.
Does compression replace a smaller top-k?
No. Top-k chooses how many candidates enter the shortlist; compression shrinks text inside those candidates. You still need an evidence token cap on context-window budgeting — and cutting k remains the first lever when the gold chunk never entered the set.
What does LongLLMLingua claim?
Microsoft’s LongLLMLingua project page reports up to a 21.4% NaturalQuestions boost with about 4× fewer tokens on GPT-3.5-Turbo, 94.0% cost reduction on LooGLE, and 1.4×–2.6× end-to-end latency improvement when compressing ~10k-token prompts at 2×–6×. Treat those as published anchors; measure on your stack.
When should you skip compression?
Skip or raise the keep-threshold when every sentence is load-bearing, when evaluation shows answer spans disappearing after compress, or when the compressor call costs more than the generator tokens you would save. Unmanaged compression can drop the rare fact that made the answer correct.