Removing Redundant Chunks Before Generation

Deduplicating retrieved context removes exact and near-duplicate chunks from the post-retrieval shortlist before packing so repeated text does not waste the evidence budget or skew the answer toward whatever was copied most. Maximal Marginal Relevance (MMR) is the sibling lever when chunks are distinct but too similar in embedding space and the query needs multi-aspect coverage. Overlap windows, mirrored FAQ pages and near-paraphrases all produce the same packing failure: you pay tokens twice and the model overweights the repeated claim. Query-time removal is the job here; ingest corpus deduplication is a different stage. The ladder is exact hash drop, then near-duplicate MinHash, then optional MMR, then compression or a lower top-k if the budget is still exceeded.

Why do retrieved chunks duplicate in RAG?

Retrieved chunks duplicate because chunk overlap re-emits shared windows on adjacent hits, because the same policy or FAQ exists in multiple sources, and because dense retrieval returns near-paraphrases that all score high on the same query. ShShell’s overlap anatomy is concrete: for example, 500-character chunks with 100-character overlap can share roughly 200 tokens when both neighbours return. That overlap knob is chosen on chunk size; this leaf only removes the copies after retrieval.

The cost is double: you pay tokens repeatedly inside the evidence slice on context window budgeting, and the model overweights the repeated claim. ShShell’s “five people, same news” framing and Caber’s duplicate-content failure notes both describe answers that lean toward whichever wording was copied most rather than toward the most authoritative source. Shelf.io, as summarised in Caber-style security write-ups, treats duplicate content as a cause of RAG errors, not a cosmetic cleanup. A dashboard that only plots retrieval recall will miss this failure until citation audits show the same paragraph id three times in one answer.

This leaf owns query-time and packing-time removal. Cleaning the corpus before embed, and incremental index hygiene, live on ingestion leaves; do not confuse those with packing dedup. Once the anatomy is clear, the mechanical question is how to drop exact and near copies while keeping the highest-ranked survivor.

Caber cites enterprise WAN and storage literature for how redundant chunks dominate real networks, including SNIA-linked ranges where a large share of storage blocks are duplicates, and points at Google and University of Pennsylvania measurements of repeated chunks across hundreds of millions of web documents. Those figures motivate why RAG prompts see the same paragraph twice; they are not a substitute for measuring duplicate rate on your own shortlist. Shelf.io’s duplicate-content error list, as Caber summarises it, is the practitioner symptom side of the same problem.

How do you deduplicate retrieved chunks?

You deduplicate retrieved chunks by running two levels on the retrieved list before packing: exact then near-duplicate, preferably after over-retrieving. Exact dedup normalizes text (lowercase, collapse whitespace) and hashes; later copies drop while the highest-ranked copy stays. Near-duplicate detection uses MinHash LSH or Jaccard similarity; How2’s default threshold of 0.85 with num_perm=128 is a common starting point, with the warning that a threshold near 0.6 can collapse distinct sections of the same concept.

The over-retrieve pattern How2 publishes is retrieve max_chunks × 3, then dedupe down to max_chunks, so recall has room before uniqueness. Keep provenance of the kept chunk so citations still resolve. Latency note from How2: building roughly 20 MinHash signatures is on the order of 5 to 10 ms; for tight SLOs run exact always and near-dup when the shortlist is large. Candidate volume before the cut is still a top-k decision. Store the Jaccard score of dropped pairs in debug metadata when a labelled answer regresses after a threshold change.

Rank-preserving keep matters: drop the later duplicate, not a random member of the cluster, so the score that put the best wording first still wins. After exact and near-dup, the remaining problem is diversity without deletion when many distinct chunks are still near-clones in embedding space.

GitHub projects such as dakshjain’s retrieval-time semantic deduplication CLI in the September 2026 organic set run cosine near-dup after retrieval and before generation, logging dedup ratio, chunk counts and token usage. That pattern matches How2’s over-retrieve-then-dedupe spine: fetch a wider top-k, drop near-copies, pack survivors. Praxen’s Medium note in the same set about fixing recall by deleting chunk rules is adjacent: overlap and chunk policy create duplicates that packing then has to remove.

What is maximal marginal relevance MMR?

Maximal Marginal Relevance (MMR) iteratively picks the next document that is relevant to the query and dissimilar to documents already selected. Carbonell and Goldstein introduced MMR in 1998. It diversifies the shortlist rather than deleting by hash. A common score form (ailog / Elastic) is MMR = λ·Sim(Di,Q) − (1−λ)·max_j Sim(Di,Dj_selected): high λ favours relevance; low λ favours diversity. The first pick is the most relevant document; then remaining candidates are re-scored.

Frameworks and stores expose the same idea with different parameter names. LangChain-style retrievers use search_type=”mmr” with fetch_k candidates returning k. OpenSearch documentation uses a diversity parameter in a form that mixes relevance and max similarity; when citing OpenSearch, say diversity closer to 1 means more diversity, and do not mix λ conventions blindly across vendors. Elastic blog ranges and Qdrant diversity-search demos are the other common references in the ranking set.

Weaviate documents diversity selection (MMR) for hybrid search as of v1.38.6 (Weaviate hybrid search documentation, September 2026): hybrid search fuses keyword and vector results, which often clusters near-duplicates at the top, and MMR reranks that fused list so each selected object adds something new. Diversity selection runs after fusion; set it on the hybrid query itself; the query limit is the candidate window and the diversity limit is how many results return; results come back in MMR order, not fused-score order. Multi-vector collections are not supported for that path. After MMR is defined, the practical question is how to set the balance parameter.

Redis and StackAI chunking guides in the organic set often discuss redundancy as a chunking problem first. Keep that depth on chunk size; this leaf only removes what retrieval already returned. MMR is the lever when you must keep k distinct aspects rather than delete near-copies of one aspect.

How do you tune the MMR diversity parameter?

You tune the MMR diversity parameter by starting relevance-leaning and moving λ only with measured redundancy. Ailog’s practitioner ranges are λ≈0.7 for general use (about 70% relevance / 30% diversity), λ=0.5 to 0.6 for high-redundancy domains, and λ=0.8 to 0.9 when precision is critical. Elastic’s blog ranges product discovery at λ=0.3 to 0.5, precision search at λ=0.7 to 0.9, and research coverage at λ=0.5 to 0.7, starting at λ=0.7 then adjusting.

OpenSearch’s default diversity of 0.5 and a candidates default around 3× size match the fetch_k ≫ k rule: MMR needs room to swap near-clones. Weaviate’s documented hybrid example uses Diversity.mmr(limit=5, balance=0.5) after a candidate limit=20 (Weaviate hybrid search documentation, as of September 2026); treat balance as Weaviate’s relevance-diversity knob and confirm the current semantics in that docs page before you copy a number into production. Always measure pairwise similarity of the packed set and answer coverage on multi-aspect queries; do not invent a universal λ. A λ that looks good on FAQ clones can demote a required twin statute on legal corpora, so retune per collection.

Cross-encoder precision after diversify is optional and belongs on cross-encoder reranking. The decision that still needs a ladder is when hash dedup is enough and when MMR should run.

When should you dedupe versus use MMR?

Use exact and near-duplicate removal when the same text or a near-copy appears twice: deletion frees tokens with no information loss. Use MMR when chunks are distinct but too similar in embedding space and you need coverage of multiple aspects: you keep k items that are not clones. The assembly ladder is exact hash drop, then near-dup MinHash, then optional MMR on survivors if multi-aspect queries still return near-clones, then contextual compression or a lower top-k if the budget is still exceeded. Do not jump to abstractive compression to “fix” duplicates that a hash pass would remove for free.

When not to run aggressive near-dup: How2’s guidance is to skip it on already-unique curated corpora where distinct sections share vocabulary. When not to run MMR: Ankit-style precision cases where every top hit is independently necessary and diversity would demote a required twin fact. Dedup at packing time is not ingest corpus deduplication (Sabarish Module 2 and similar ingest guides); that work happens before embed.

Permission leakage from shared duplicate chunks is a security concern Caber-style write-ups raise; route that failure to multi-tenant leakage rather than treating it as a packing tweak. After the ladder is chosen, implementation is wiring the steps after retrieval. Teams that jump straight to MMR without an exact-hash pass usually discover they were paying for identical strings under two UUIDs.

How do you implement retrieval deduplication?

You implement retrieval deduplication by running exact then near-dup then optional MMR after retrieval and optional rerank, then packing under the evidence token cap. Pin versions and show outputs on pipeline build. The parent stage is building the prompt the model actually sees.

Weaviate leads product lists for diversity-aware hybrid retrieval: hybrid search with diversity selection (MMR) from v1.38.6, as documented on docs.weaviate.io (September 2026), then LangChain mmr retrievers, OpenSearch MMR extensions and Qdrant diversity search as other common APIs. Ordering after uniqueness is a sibling step on context ordering. Keep the kept-chunk provenance so generation can cite the survivor rather than a dropped duplicate. Log how many chunks each ladder step removed for at least one full week of production traffic so λ and Jaccard thresholds move on measured redundancy rather than a default copied from a vendor blog post.

Deduplicating retrieved context stops repeated text from wasting the evidence budget and skewing the answer. Exact hash and MinHash near-dup delete copies; MMR diversifies a shortlist when multi-aspect coverage needs non-clones. Weaviate’s hybrid MMR path (v1.38.6+) is the default store example when products are named together; other frameworks expose the same Carbonell and Goldstein idea under different parameter names. Run the ladder in order, measure redundancy and coverage, and only then compress or cut k. Assembly owns the packing decision; ingestion owns cleaning the corpus. A shortlist that is unique and diverse still has to fit the evidence cap before generation runs. Measure duplicate rate every production week or the ladder drifts back to defaults. Caber’s chunk-level governance framing is a reminder that duplicates are not only a token tax; they can also recombine permissioned fragments in ways document-level ACLs never intended, which is why leakage belongs on its own failure leaf when tenants share near-identical text across collections in multi-tenant vector stores. That closes the packing ladder before generation runs.