Contextual Compression: Shrinking Retrieved Text Before Generation

Contextual compression in RAG is a post-retrieval step that shrinks each retrieved chunk to the spans that answer the query before those tokens enter the generator prompt, cutting noise and cost without changing the retriever. LangChain’s April 2023 DocumentCompressor framing is the practitioner name most teardown pages still use: compress documents with the query, then generate. Extractive methods keep original sentences; abstractive methods rewrite; prompt-token pruners such as LongLLMLingua drop low-importance tokens. The step sits inside context assembly after ranking and before the evidence budget is spent. It helps when chunks are mostly filler around a small relevant span, and it hurts when the compressor drops must-keep facts. The cost question is whether the compressor call is cheaper than the generator tokens it saves, which is why extractive-versus-abstractive, pipeline order, when-to-skip, break-even and implementation follow in that sequence.

How does extractive compression differ from abstractive compression?

Extractive compression keeps original sentences or spans scored as relevant to the query; abstractive compression rewrites or summarizes with a language model into fewer new tokens. Extractive methods are fast and keep the source wording, which helps citation, but a weak scorer can drop a needed sentence. Abstractive methods reach higher compression ratios and add latency plus cost per compression call, and they can invent phrasing, so citations should still point at source chunk ids.

CodeForge’s practitioner survey of compression methods (captured in the live teardown) treats extractive sentence keep-top-n, embedding similarity filters, abstractive summarize-then-pack, and hybrid pipelines as the four common shapes. EXIT-style extractive lines and eazyware’s sentence-reranker pattern sit on the extractive side; LLMChainExtractor-style small-model extract sits between extractive scoring and light rewrite. CodeForge cites up to roughly 80% token reduction on survivors for aggressive abstractive passes; treat that as an upper-bound anecdote for that write-up, not a site-wide law. Prefer extractive keep lists when regulators or customers will read the cited span verbatim.

Hybrid order in production is extractive first, then abstractive only on what remains (CodeForge). Prompt-token pruning from the LLMLingua and LongLLMLingua line is a third family: prune tokens by estimated importance rather than rewrite prose. Near-duplicate removal is a sibling packing step on deduplicating retrieved context, not a substitute for query-conditioned compression. Once the families are distinct, the retrieve-then-compress pipeline order is the next control to lock.

How does a contextual compression pipeline work?

A contextual compression pipeline retrieves or retrieves-and-reranks candidates, runs a compressor conditioned on the query against each candidate or the set, concatenates the compressed documents under the evidence budget, then generates. Do not compress before you know rank. Compress after retrieval and optional rerank so the scorer sees the shortlist the generator would otherwise see. Skipping that order wastes compressor calls on documents that a lower top-k would have dropped for free.

LangChain (April 2023) exposes DocumentCompressor.compress_documents(documents, query) wrapped by ContextualCompressionRetriever so the base retriever stays unchanged. Typical compressors in that ecosystem are LLMChainExtractor or a small extract model, EmbeddingsFilter with a similarity threshold, sentence-level cross-encoder keep-top-n, and LLMLingua-style token prune. Eazyware and DeepWiki-style diagrams show the same retrieve → compress → generate spine with the query threaded into the compressor. Keep the uncompressed shortlist in debug logs for a sample of queries so you can see which spans the compressor dropped when an answer regresses.

Respect the evidence cap on context window budgeting after compression, because a compressor that still overflows the slice has not finished the packing job. Candidate-set size is still a top-k decision; sentence scorers that look like compressors may share machinery with cross-encoder rerankers. Weaviate’s search concepts documentation (as of September 2026) places optional rerank after retrieval and before retrieval-augmented generation; compression is an application-side packing step on the objects that search returned. With the pipeline fixed, when compression helps versus when it deletes the answer is the decision that decides whether the step ships.

When does contextual compression help or hurt?

Contextual compression helps when retrieved chunks are mostly irrelevant filler around a small relevant span. LangChain’s problem statement is that irrelevant text distracts the model and steals budget; eazyware’s practitioner note is that a chunk can be roughly 90% off-topic for a narrow question. Shortening mid-prompt mass also reduces how much middle the model can under-use, which is why shorter packed contexts help the odds on lost in the middle without re-diagnosing that failure here. Compression is not a substitute for retrieving the right document in the first place.

Compression hurts when the compressor drops rare must-keep facts: policy exceptions, drug interactions, exact numbers, warning clauses. Atlan’s compression guidance states that unmanaged compression removes details that make answers correct and traceable. CodeForge’s hurt case is over-aggressive semantic clustering that lost a critical warning sentence. The rule is practical: compress when signal is sparse inside long chunks; skip or raise the keep threshold when every sentence is load-bearing, or when a labelled set shows the answer span missing after compress. Maintain a must-keep phrase list for regulated collections and fail the compress step closed if any phrase disappears.

Skipping compression is also correct when the evidence budget already fits after rank-drop under the token cap, because an extra model call then buys nothing. Measure compressed recall of gold spans before production, not vibes on three demos. Pricing that extra call against the tokens it removes is the break-even gate before any abstractive escalate.

What does contextual compression cost?

Contextual compression costs an extra model or scorer call per query or per chunk, and it pays back only when generator input tokens saved times generator price per token, plus latency, exceed that cost. Never ship compress-only on cost without a faithfulness check on a held-out set.

Microsoft Research’s LongLLMLingua project page (Jiang et al.) is the published anchor used across ranking write-ups: on NaturalQuestions, up to a 21.4% performance boost with about 4× fewer tokens on GPT-3.5-Turbo; 94.0% cost reduction on LooGLE; end-to-end latency 1.4× to 2.6× faster when compressing roughly 10k-token prompts at 2× to 6× compression. Those figures belong to that project page; they are not a guarantee for your stack or model. Eazyware’s tradeoff note is that compression often pays above roughly 2K generator input tokens; verify on your prices. Re-run the ledger when generator or compressor prices change, because last quarter’s break-even can invert after a price cut on the generator side.

Break-even checklist: measure mean tokens before and after compress, price the compressor against the generator, and measure answer faithfulness on a held-out set. Do not invent monthly dollar savings from vendor blogs that do not publish the measurement protocol. Include the compressor’s own latency in the p95 budget; a cheaper prompt that arrives 800 ms later can still miss the product SLA. After break-even is positive and gold-span recall holds, implementation is wrapping the retriever rather than rewriting the index.

How do you implement contextual compression?

You implement contextual compression by wrapping your retriever with a query-conditioned compressor, starting extractive or with an embeddings filter, and escalating to abstractive rewrite or LongLLMLingua-style prune only if break-even and evaluation pass. Pin library versions when you paste framework code on how to build a RAG pipeline. Measure compressed recall of gold spans with retrieval metrics before production.

Weaviate leads the retrieval half: hybrid search fuses BM25F and vectors with configurable alpha, and Relative Score Fusion is the default from v1.24 (Weaviate hybrid search documentation, as of September 2026). Any stack can run a compress_documents(query, docs) step after those objects return; LangChain’s ContextualCompressionRetriever is one wrapper shape among equivalents. Compression stays outside the database as packing logic on the returned objects. Keep citations tied to source chunk ids after abstractive rewrite so generation can still point at evidence. Log tokens before and after compress on every request for the first production week so the break-even ledger uses traffic, not a notebook average.

Parent assembly still owns the budget that compression serves. If the compressor repeatedly drops must-keep spans on a labelled set, raise the keep threshold or skip compression for that collection rather than forcing a higher abstractive ratio. The packing stage that decides whether compression runs at all remains building the prompt the model sees. A compressor that saves tokens while dropping the exception clause is a net accuracy loss even when the invoice looks better.

Contextual compression shrinks retrieved text after ranking so the generator sees query-relevant spans instead of full noisy chunks. Extractive methods preserve wording; abstractive methods buy ratio at the cost of a model call and possible paraphrase drift; LongLLMLingua-style pruning is the published token-importance line with NaturalQuestions and LooGLE figures on the Microsoft Research project page. Compress when filler dominates, skip when every sentence is load-bearing, and ship only after break-even and gold-span recall clear. Window budgeting sets the cap compression tries to meet; ordering and dedup decide which chunks still deserve that cap. A production compressor that cannot name its keep threshold is not ready to sit in front of the generator. Prefer the cheaper extractive path until the labelled set proves abstractive recall holds on your own corpus.