Cleaning Documents Before You Chunk Them

Document cleaning for RAG strips boilerplate and chrome — headers, footers, navigation, ads, cookie banners, repeated disclaimers, page numbers — from parsed text so only answer-bearing content is chunked and embedded. It sits after parse and before the chunker on the ingest path. This page covers why uncleaned text breaks retrieval, what to strip, how to remove it, what to keep, and where the runnable glue lives. The stage map is document ingestion; what happens after a clean hand-off is chunking.

Why does uncleaned document text break RAG retrieval?

Uncleaned document text breaks RAG retrieval because anything left in the parse is treated as content by the chunker and the embedder — chrome tokens become passages that compete with real answers.

Three concrete failures show up when you skip the strip step (Drive AI’s clean-ingestion framing on the live 2026-07-27 SERP for clean documents for rag; SearchCans and Teammately Docs state the same chain):

• Token waste. Navigation, footers, and template disclaimers fill chunks and the context window without answering the query. Drive AI’s tutorial notes that a short article pulled from raw HTML can balloon into thousands of noise tokens; treat that magnitude as a vendor illustration and measure the delta on your own pages.
• Diluted embeddings. When a chunk mixes body prose with menu labels and running headers, the vector represents chrome alongside meaning — relevant passages score lower in similarity search than they should.
• Hallucination pressure. Teammately’s document-cleaning docs put it bluntly: noisy source content leads to inaccurate responses because the model has to invent where the retrieved context is junk.

For web-scale HTML dumps, Diomgis (text-extraction / boilerplate note) cites a Common Crawl figure: roughly 40–50% of tokens in a naively extracted dump are navigation, cookie banners, footer links, legal chrome, and ads. That is a pretraining-corpus magnitude, not a published RAG product benchmark — use it as an order-of-magnitude warning for HTML ingestion, not as your retrieval KPI. SearchCans (January–April 2026) claims raw scrapes are 60–70% noise and that pre-cleaning can cut LLM token cost by “up to 30%”; those are vendor claims — verify on your corpus before you budget against them.

What should you strip before chunking?

You should strip the chrome classes that become false passages if they reach the index — and leave structure and answer text alone.

The removal list that recurs across ChatNexus’s data-preparation guide, Databricks’ unstructured pipeline (“Data cleaning”), SearchCans’ web-noise table, and bthek1’s preprocessing notes is:

• Running headers and footers — titles and legal lines repeated on every page.
• Page numbers — “Page 3 of 12” lines that add nothing to meaning.
• Navigation menus and sidebars — category links, account chrome, site maps.
• Cookie and consent banners — ephemeral UI, not knowledge.
• Advertisements and promo blocks — commercial copy that pollutes embeddings.
• Template disclaimers — the same confidentiality footer on every document, not a one-off operative clause.
• Scripts, styles, and raw HTML tags — presentation markup, not prose.
• OCR gibberish lines — runs with extreme non-letter ratios after a bad scan (detect here; the reject-versus-index policy for whole scanned pages lives on OCR and scanned documents).

If headers and footers were interleaved into body text by a weak PDF extractor, fix reading order on PDF parsing for RAG first — cleaning removes chrome from a parse string; it does not reconstruct columns. Flattened tables are not “boilerplate” either: keep structure via table ingestion, do not strip the cells.

How do you remove boilerplate from documents for RAG?

Boilerplate removal for RAG uses three strategy families — frequency heuristics, structural or DOM rules, and LLM-prompted strips — then normalises what remains so the chunker sees stable text.

1. Frequency heuristic. Lines that appear in a large share of documents in the corpus are usually chrome. bthek1’s preprocessing guide states a practical starting rule: lines in more than about 80% of documents are likely boilerplate. That percentage is a method knob, not a published F1 on your data — tune it.
2. Structural / DOM rules. For HTML, drop nav, header, footer, script, and style nodes, then run a main-content extractor. Diomgis compares the classic families: density scorers (Boilerpipe-class), tree-based scorers (Trafilatura-class, used in Hugging Face’s DataTrove pipeline), and Readability-style algorithms (high precision, more conservative recall). For PDFs, detect text that repeats at fixed page positions (bthek1’s structural strategy).
3. LLM-prompted cleaning. Ask a model to mark headers, footers, and disclaimers, then delete those spans. Use when heuristics miss template-heavy enterprise docs; budget the per-document cost — this is the expensive path at corpus scale.

After removal, normalise: collapse runaway whitespace, apply Unicode NFKC so compatibility characters do not invent duplicate tokens (bthek1; Diomgis), and tame OCR artefact punctuation runs. On the web path, stop at clean Markdown — DEV’s HTML-versus-Markdown note and SearchCans’ Reader framing both treat Markdown as the AI-friendly intermediate so structure-aware chunking can split on real headings instead of layout debris.

What should you keep when cleaning documents for RAG?

Document cleaning for RAG is not maximal deletion: keep every line a competent reader would need to answer, and strip only strings that repeat as chrome across pages or that are pure UI.

Ranking guides on the top-ranking results list what to remove; almost none state the keep rule. Use this decision:

• Keep section titles and headings — they are retrieval signal and the boundaries structure-aware splitters need.
• Keep figure and table captions — they name the entity the numbers belong to.
• Keep footnotes and caveats attached to the claim they qualify — stripping them orphans the fact.
• Keep operative legal or policy clauses even when they look “boilerplate” — if the clause is the answer surface (refund rules, liability limits), it is content the first time it appears; strip only the identical footer stamped on every page.
• Keep short FAQs and definition lines — density heuristics sometimes delete them because they are brief.

Over-stripping shows up as missing the document you know is there: the source was ingested, but the answer sentence never survived cleaning. Under-stripping shows up as chrome winning similarity — a flavour of wrong chunk where the “passage” is a navigation label.

How do you implement document cleaning before chunking?

Implement document cleaning as a hard stage after a successful parse and before any splitter, with a quality gate that rejects empty or garbage text before you pay for embeddings.

A defensible minimum from the live teardown (bthek1 quality-gate pattern; Databricks’ “inspect the parse” culture on the sibling PDF leaf):

• Order. Parse → clean → enrich metadata → chunk → embed. Do not chunk first and “clean later” — chrome already owns vectors by then.
• Gate. Drop units below a minimum length (bthek1 starts around 50 characters) and flag text whose letter ratio is extremely low (bthek1’s example floor is about 0.4) as likely OCR or extract garbage. Both numbers are starting heuristics — tune them on your corpus; they are not universal laws.
• Inspect. Spot-check cleaned pages the way you spot-check parses: if headers still appear mid-paragraph, the strip rules failed.

Runnable, pinned pipeline code belongs on building a RAG pipeline from scratch. The ingest stage map is document ingestion. After the text is clean, choose a splitter on chunking, and attach source, page, and ACL fields on metadata extraction — cleaning does not replace those leaves.