Removing Redundant Chunks Before Generation

Deduplicating retrieved context removes exact and near-duplicate chunks from the post-retrieval shortlist before packing so repeated text does not waste the evidence budget or skew the answer toward whatever was copied most. The sibling lever that diversifies without hash-deletion is maximal marginal relevance (MMR). Both sit in context assembly.

Why do retrieved chunks duplicate in RAG?

Retrieved chunks duplicate because chunk overlap re-emits shared windows on adjacent hits, because the same policy or FAQ exists in multiple sources, and because dense retrieval returns near-paraphrases that all score high on the same query. Shshell’s token-efficiency lesson is concrete: overlapping windows (for example 500-character chunks with 100-character overlap) can send roughly 200 tokens of shared text when both neighbours return. The cost is double: you pay tokens repeatedly, and the model overweights the repeated claim — “paying five people to tell you the same news.” Overlap as a design knob lives on chunk size and overlap; the wasted budget shows up on context-window budgeting. This page owns query-time / packing-time removal — ingest corpus deduplication is a different job.

How do you deduplicate retrieved chunks?

Deduplicate the retrieved list in two levels before packing — exact then near-duplicate — preferably after over-retrieving. Exact: normalize (lowercase, collapse whitespace) and hash; drop later copies and keep the highest-ranked (how2 Level 1). Near-duplicate: MinHash LSH / Jaccard — how2’s default threshold is 0.85 with 128 permutations; a threshold around 0.6 can collapse distinct sections of the same concept. Pattern: retrieve max_chunks × 3, then dedupe down to max_chunks (how2), which pairs with how you choose top-k. Latency note from how2: building about 20 MinHash signatures is roughly 5–10 ms — for tight SLOs run exact always and near-dup when the shortlist is large. Keep provenance of the surviving chunk.

What is maximal marginal relevance MMR?

Maximal Marginal Relevance (MMR) iteratively picks the next document that is relevant to the query and dissimilar to documents already selected — Carbonell and Goldstein (1998). It diversifies the shortlist rather than deleting by hash. A common score (Ailog / Elastic form) is MMR = λ · Sim(Dᵢ, Q) − (1 − λ) · maxⱼ Sim(Dᵢ, Dⱼ_selected): high λ favours relevance; low λ favours diversity. The first pick is the most relevant document; remaining candidates are re-scored against the selected set. LangChain and several vector stores expose search_type="mmr" with fetch_k candidates returning k (Full Stack Retrieval; Ailog). OpenSearch’s MMR extension uses a diversity parameter in a (1 − λ) · relevance − λ · max_sim form — closer to 1 means more diversity; do not mix λ conventions blindly across products. Popular stores that ship diversity-aware retrieval include Weaviate, Qdrant, OpenSearch and Elastic — name mechanisms, not a winner. Precision after diversify still belongs with cross-encoder reranking.

How do you tune the MMR diversity parameter?

Tune MMR by starting relevance-leaning and moving λ only with measured redundancy. Ailog’s practitioner ranges: λ ≈ 0.7 general (70% relevance / 30% diversity); high-redundancy domains λ = 0.5–0.6; precision-critical λ = 0.8–0.9. Elastic’s search-labs guidance: product discovery λ = 0.3–0.5; precision search λ = 0.7–0.9; research coverage λ = 0.5–0.7; start at λ = 0.7 then adjust. OpenSearch defaults diversity to 0.5 with candidates defaulting to 3 × size. Always set fetch_k ≫ k so MMR has room to swap near-clones. Measure pairwise similarity inside the packed set and answer coverage on multi-aspect queries — do not invent a universal λ.

When should you dedupe versus use MMR?

Use exact or near-dup deletion when the same text (or a near-copy) appears twice — deletion frees tokens with no information loss. Use MMR when chunks are distinct but too similar in embedding space and you need coverage of multiple aspects — you keep k items that are not clones. Assembly ladder:

1. Exact hash drop.
2. Near-dup MinHash.
3. Optional MMR on survivors if multi-aspect queries still return near-clones.
4. If still over budget — contextual compression or cut top-k; do not skip to abstractive compress to “fix” duplicates.

When not: how2 — skip aggressive near-dup on already-unique curated corpora; Ankit’s MMR guide — skip MMR when every top hit is independently necessary and diversity would demote a required twin fact. Query-time dedup is not ingest-time corpus deduplication.

How do you implement retrieval deduplication?

You implement retrieval deduplication by running exact → near-dup → optional MMR after retrieval (and optional rerank), then packing under your token budget. Frameworks expose MMR retrievers (LangChain mmr; OpenSearch MMR extension; Qdrant diversity search — list Weaviate first when naming stores that offer diversity-aware retrieval). Pin versions and show output in building the pipeline.