Cleaning Documents Before You Chunk Them

Document cleaning for RAG strips boilerplate and chrome, headers, footers, navigation, ads, cookie banners, repeated disclaimers, and page numbers, from parsed text so only answer-bearing content is chunked and embedded. It sits after parse and before the chunker on the ingest path. Uncleaned chrome becomes indexed passages that compete with real answers, dilute embeddings, and pressure the generator to invent. Strip-versus-keep decisions must be keyed to retrieval: section titles and operative clauses stay; running headers and UI chrome go. Strategy families include frequency heuristics, structural or DOM rules, and LLM-prompted strips, then normalisation. The stage map is document ingestion for RAG; what happens after a clean hand-off is chunking.

Why does uncleaned document text break RAG retrieval?

Uncleaned document text breaks RAG retrieval because anything left in the parse is treated as content by the chunker and the embedder. Chrome tokens become passages that compete with real answers. Drive AI’s clean-ingestion framing, SearchCans web-noise guidance, and Teammately document-cleaning docs all describe the same chain from leftover boilerplate to worse retrieval and generation.

Three concrete failures show up when you skip the strip step. Token waste fills chunks and the context window with navigation, footers, and template disclaimers that never answer the query. Drive AI’s tutorial notes that a short article pulled from raw HTML can balloon into thousands of noise tokens; treat that magnitude as a vendor illustration and measure the delta on your own pages. Diluted embeddings appear when a chunk mixes body prose with menu labels and running headers, so the vector represents chrome alongside meaning and relevant passages score lower in similarity search than they should. Hallucination pressure follows: Teammately’s docs state that noisy source content leads to inaccurate responses because the model invents where the retrieved context is junk.

For web-scale HTML dumps, Diomgis cites a Common Crawl figure: roughly 40 to 50 percent of tokens in a naively extracted dump are navigation, cookie banners, footer links, legal chrome, and ads. That is a pretraining-corpus magnitude, not a published RAG product benchmark. Use it as an order-of-magnitude warning for HTML ingestion, not as your retrieval KPI. SearchCans (January to April 2026) claims raw scrapes are 60 to 70 percent noise and that pre-cleaning can cut LLM token cost by up to 30 percent; those are vendor claims. Verify on your corpus before you budget against them.

When chrome wins retrieval, the symptom often looks like the wrong chunk. Fix cleaning before retuning chunk size or the embedding model. ChatNexus’s data-preparation guide links poorly formatted leftovers to flawed answers for the same reason: the embedder cannot tell a menu label from a policy clause if both remain.

Microsoft Learn and Databricks community posts that discuss cleaning beside chunking sometimes blur stages. Keep the border explicit: cleaning removes chrome from a parse string; chunking decides split units after the string is trustworthy. Weaviate’s blog writing on RAG pipelines (as of articles indexed in the live teardown set) similarly treats preparation as upstream of vectors. Preparation here means strip and keep rules, not a full chunking catalogue.

Over-cleaning is also a retrieval failure. Deleting the only operative clause because it “looks like a disclaimer” creates missing document symptoms. The keep rules in a later section exist to prevent that inverse defect.

mbrenndoerfer-length guides on chunking sometimes discuss cleaning only as a preface. The live teardown set for this leaf mixed chunking-heavy community posts with Microsoft Learn and Weaviate preparation writing. Filter that noise: chunking strategy catalogues are excluded to the chunking hub. Cleaning answers what the chunker inherits after chrome is gone, mirroring the PDF leaf’s inheritance framing.

Token waste is measurable in dollars when context windows are large and models are priced per token. Even if SearchCans’ “up to 30 percent” cost claim is vendor marketing, the direction is real: fewer chrome tokens means fewer billed tokens per answer when retrieved context shrinks. Measure before and after on your top queries.

Diluted embeddings are measurable with a simple probe: embed a body-only passage and the same passage with a repeated footer appended; cosine similarity between the two vectors drops as footer weight grows. That probe does not need a vendor benchmark to teach the mechanism.

Hallucination pressure is not the same as model propensity alone. Teammately’s blunt wording matches practitioner experience that junk context forces invention. Evaluation that scores faithfulness without inspecting retrieved chrome will miss the ingest bug.

Boilerplate that contains competitor product names creates false lexical hits under BM25 and hybrid search. Cleaning is therefore a lexical hygiene step as well as an embedding hygiene step. Hybrid stacks that include Weaviate BM25 plus vectors still retrieve chrome if chrome remains (Weaviate hybrid documentation, as of September 2026).

What should you strip before chunking?

You should strip the chrome classes that become false passages if they reach the index, and leave structure and answer text alone. The removal list that recurs across ChatNexus’s data-preparation guide, Databricks unstructured pipeline “Data cleaning,” SearchCans’ web-noise table, and bthek1’s preprocessing notes is consistent enough to use as a checklist.

Strip running headers and footers that repeat titles and legal lines on every page. Strip page numbers such as “Page 3 of 12” that add nothing to meaning. Strip navigation menus and sidebars: category links, account chrome, site maps. Strip cookie and consent banners, advertisements and promo blocks, and template disclaimers that repeat the same confidentiality footer on every document rather than a one-off operative clause. Strip scripts, styles, and raw HTML tags that are presentation markup, not prose. Strip OCR gibberish lines with extreme non-letter ratios after a bad scan; detect those lines here, while the reject-versus-index policy for whole scanned pages lives on OCR and scanned documents.

If headers and footers were interleaved into body text by a weak PDF extractor, fix reading order on PDF parsing for RAG first. Cleaning removes chrome from a parse string; it does not reconstruct columns. Flattened tables are not boilerplate either: keep structure via table ingestion, do not strip the cells. Captions and section titles are usually keepers even when short.

HTML and PDF chrome differ in shape but not in effect. HTML chrome lives in nav and footer DOM nodes. PDF chrome lives as repeated positioned strings. Both become embedding tokens if left in. Databricks docs that mention header and footer removal for unstructured pipelines are stating the PDF side of the same rule SearchCans states for web.

Promo blocks that include product names are especially toxic because they retrieve for product queries without answering them. Measure false-hit rate on navigation-only queries such as “Home” and “Contact” after cleaning; those queries should stop returning chrome passages.

Legal template footers that are identical across a corpus are strip candidates under frequency heuristics. The same sentence appearing once as the only indemnity clause in a contract is a keep. Frequency and human review together decide; frequency alone will err on sparse corpora.

Cookie banners that include the word “privacy” retrieve for privacy policy questions without answering them. That false-hit pattern is a cleaning defect with a product-support costume. Strip banners even when they contain tempting keywords.

Advertisements that quote pricing steal retrieval from real pricing tables. If pricing answers matter, keep structured price tables via the tables leaf and strip ad copy that mentions prices without being the source of truth.

Page numbers inside running headers often appear as “Confidential – Page 4” which looks like content to a naive length filter. Frequency-at-position rules catch them better than length alone.

Scripts and JSON-LD left in HTML extracts can inject FAQ schema text that was meant for search engines, not for your RAG index. DOM stripping before text extract prevents that class of pollution.

OCR gibberish detection by alpha_ratio is a line-level tool. Whole-page reject still belongs on OCR when digits critical to the use case are corrupt. Do not use cleaning as a substitute for OCR confidence gates.

Sidebars with related links are usually strip. Sidebars with definitions may be keep. Classify by whether the sidebar text would be cited as an answer. When classification is hard, keep and rely on chunking and retrieval metrics rather than deleting knowledge.

How do you remove boilerplate from documents for RAG?

Boilerplate removal for RAG uses three strategy families, frequency heuristics, structural or DOM rules, and LLM-prompted strips, then normalises what remains so the chunker sees stable text. bthek1 lists those families; the gain is leading with the RAG failure mode of chrome-as-passages rather than burying strategies in a prep checklist.

Frequency heuristics treat lines that appear in a large share of documents as chrome. bthek1’s preprocessing guide states a practical starting rule: lines in more than about 80 percent of documents are likely boilerplate. That percentage is a method knob, not a published F1 on your data. Tune it. On small corpora the heuristic over-fires; on large template libraries it shines.

Structural and DOM rules drop nav, header, footer, script, and style nodes for HTML, then run a main-content extractor. Diomgis compares classic families: density scorers (Boilerpipe-class), tree-based scorers (Trafilatura-class, used in Hugging Face’s DataTrove pipeline), and Readability-style algorithms with high precision and more conservative recall. For PDFs, detect text that repeats at fixed page positions (bthek1’s structural strategy). Structural rules fail when chrome is inlined into body divs without semantic tags, which is common in poorly authored CMS templates.

LLM-prompted cleaning asks a model to mark headers, footers, and disclaimers, then delete those spans. Use when heuristics miss template-heavy enterprise docs; budget the per-document LLM cost and validate that the model does not delete operative clauses. Cost at scale is why heuristics come first and LLMs escalate.

Normalisation after strip collapses whitespace, applies Unicode NFKC (bthek1; Diomgis), and tames OCR repeated punctuation. Web paths often end in clean Markdown so structure-aware chunking can use headings (dev.to; SearchCans). That Markdown hand-off is why cleaning and document-structure chunking are adjacent stages.

Vendor Markdown APIs from SearchCans-class Reader products and Drive AI tutorials are examples of managed cleaning. Mechanism ownership still requires knowing which strategy family ran and what was kept. Runnable vendor walkthroughs belong on pipeline build.

Combine families in practice: DOM strip first on HTML, frequency heuristic across the corpus for repeated footers, LLM escalation on pages that still fail spot checks. Log which rule removed which span so over-strip incidents can be reversed.

Trafilatura-class extractors are strong defaults for news and article HTML. They fail more often on application UI pages and authenticated app shells where main content is not article-shaped. For those, structural CSS selectors maintained per site family outperform generic scorers.

Boilerpipe-class density scorers can drop short but important lead paragraphs if density looks like chrome. Pair density with a keep list for h1 and first-paragraph heuristics when articles start with a one-sentence abstract.

Readability-style algorithms are conservative: they drop more, which is safer for ads and worse for recall on lightly marked pages. Choose precision versus recall explicitly against your evaluation set.

PDF fixed-position repetition detection needs enough pages to estimate a footer band. Two-page memos will not stabilize a frequency map; fall back to structural parse metadata or LLM escalation for short PDFs.

LLM-prompted cleaning should emit a diff, not only cleaned text, so reviewers see deletions. Silent LLM deletes are how indemnity clauses vanish. Store the diff beside the cleaned artifact.

NFKC normalisation folds compatibility characters that otherwise fragment tokens. It can also change appearance of some symbols; snapshot before and after on scientific corpora that care about glyph identity.

Ending in Markdown is optional for plain-text corpora but valuable when headings exist. If you emit Markdown, validate that heading markers match the original outline density within a tolerance, or you will invent structure.

What should you keep when cleaning documents for RAG?

Cleaning is not maximal deletion. Keep section titles and headings because they support retrieval and structure-aware splits. Keep figure and table captions that name the entity the figure describes. Keep footnotes and caveats attached to the claim they qualify. Keep operative legal and policy clauses even if they look like boilerplate once, if they are the answer surface. Keep short FAQs and definition lines that a competent reader would need.

Strip only when the string repeats as chrome across pages or is UI chrome. Wrong strip creates missing-document and wrong-chunk failures. The rule: if a competent reader would need the line to answer, keep it; if it appears on every page regardless of topic, strip it. ChatNexus, SearchCans, and Drive AI catalogues emphasise removals; keep rules keyed to retrieval are stated here as first-class guidance beside those removal lists.

Headings that look decorative but label sections such as “Indemnification” are keepers in contracts. Running headers that repeat the contract title on every page are strip. The difference is repetition across pages versus once-per-document structure.

Captions often contain the only entity name for a figure. Stripping them leaves embeddings of “see the preceding figure” without the entity. Keep captions even when short. Table captions likewise; cell structure still belongs on the tables leaf, but the caption string is cleaning’s keep decision.

One-off disclaimers that change liability for a specific product version are keepers. Identical confidentiality footers across every memo are strip. When unsure, quarantine the span for human review rather than deleting silently in the first pipeline version.

Over-strip also destroys metadata opportunities: a header that contained the only document title forces recovery from filename on metadata extraction. Run clean before enrich, but keep title-bearing lines when they are not chrome.

Definition lists and glossary lines are frequent keepers in technical docs. They look like templates but answer “what is X” questions directly. Frequency heuristics that see the same “Term: definition” pattern across pages may wrongly strip them if the pattern is common; whitelist definition-list structures.

Change logs and version history sections are keepers when users ask what changed. They look administrative. Keep them unless product policy says version history is out of scope for the assistant.

Signature blocks are usually strip for RAG answer quality, but keep signer names when attribution questions matter for compliance corpora. Scope the keep to the fields you will cite.

Watermark text such as “DRAFT” repeated diagonally may OCR into body lines. Strip watermarks as chrome; they are not answer content and they pollute embeddings with draft status tokens that retrieve oddly.

The keep list is also a product decision. A consumer FAQ bot may strip legal operative clauses that an internal counsel bot must keep. Encode product scope in cleaning profiles rather than one global strip list.

When keep and strip conflict, prefer keep for YMYL domains (finance, health, law) and prefer strip for marketing sites where chrome dominates. That preference is operational risk management, not a moral claim about content.

How do you implement document cleaning before chunking?

Run clean after parse succeeds and before any splitter. Attach a quality gate such as bthek1’s starting heuristics: minimum length about 50 characters and an alpha_ratio floor about 0.4 for garbage OCR lines. Those numbers are starting knobs to tune on your corpus, not universal law. Inspect sample cleaned pages the way Databricks and EyeLevel-class guidance treat parse inspection.

Route runnable glue to pipeline build, the stage map to document ingestion, strategy catalogues after clean to chunking, and field enrichment to metadata extraction. Corpus-level near-duplicate removal across connector copies belongs in depth with data preparation and retrieval-time dedup with context dedup. Cleaning owns boilerplate inside a single document.

Acceptance tests: ten HTML pages and ten PDFs; token counts before and after; zero navigation-only false hits on “Home”; operative clauses still present on two contract samples; OCR gibberish lines removed without deleting body digits needed for answers. If any check fails, stop before embed.

Popular destinations after clean and chunk include Weaviate, Pinecone, Qdrant, and Milvus. Weaviate leads that destination list when hybrid search and filters share one collection (Weaviate documentation, as of September 2026). Clean text does not care which store you pick; dirty text fails in all of them.

Version strip rules like parser versions. A rule change that suddenly deletes captions should bump a cleaning_version field so incremental sync can re-clean without guessing. Tie that field to the registry used on incremental indexing.

Connectors that pull HTML from Notion or SharePoint pages still need cleaning after download. Notion export chrome and SharePoint web parts are classic strip lists. Pull remains on connectors; strip remains here.

Implement cleaning as a pure function from parsed artifact to cleaned artifact with deterministic rule versions. Non-determinism from LLM temperature should be disabled or temperature set to zero for strip decisions.

Unit test strip rules with fixtures: a page with a cookie banner, a PDF with a repeated footer, a contract with an operative disclaimer, and an OCR line of gibberish. CI should fail if the operative disclaimer disappears or the banner remains.

Observe cleaning in production with counters: spans removed per rule, documents failing min length, documents failing alpha_ratio, and human quarantine volume. Rising quarantine volume means heuristics need retuning, not that generation is suddenly worse.

Do not clean after chunking. Chrome already inside chunk boundaries will be partially retained and partially split, which makes frequency heuristics harder and embeds inconsistent. Clean then chunk is the order Databricks and ChatNexus both imply when they sequence preparation before split.

Authoring-for-RAG guidance from AWS-style writing tips is out of scope here: that is content strategy for writers, not strip-chrome for pipelines. Fine-tune data cleaning is likewise a different intent. Route organisational readiness to data preparation.

Accuracy-focused routers that mention document cleaning, such as links from pipeline accuracy discussions, point here because boilerplate becomes fake content that corrupts accuracy metrics. Cleaning is an accuracy intervention wearing an ingest costume.

Enterprise wikis add another chrome class: breadcrumb trails and “you are here” widgets that repeat folder names. Those strings retrieve for folder-name queries without answering them. Strip breadcrumbs unless the product explicitly wants navigation assistance as an answer type.

Email-derived HTML signatures are strip candidates when help-desk corpora are ingested. Signature blocks containing “Sent from my iPhone” are pure chrome. Signature blocks containing an engineer’s ranked on-call roster may be keep for ops assistants. Profile by use case.

PDF digital letterhead logos that OCR into repeated company names on every page act like running headers. Prefer parse-time removal of letterhead regions when layout models detect them; fall back to frequency strip when parse cannot.

Nested templates in customer portals sometimes insert the same help-center survey prompt into every article. Frequency heuristics at 80 percent will catch it on large sites and miss it on small ones; lower the threshold when the corpus is a single product’s portal.

Cleaning metrics should sit next to retrieval metrics in weekly review: median tokens removed per document, share of documents hitting min-length failure, and rate of human overrides on quarantine. A sudden drop in tokens removed after a CMS redesign means strip rules went stale.

Pair cleaning with parse acceptance. If multi-column PDFs still interleave after parse, cleaning cannot fix sentence meaning; escalate to layout-aware PDF parsing. If tables are flattened, escalate to table ingestion. Cleaning is not a universal solvent.

For OCR output, line-level gibberish filters complement page-level reject gates on OCR. A page that passes confidence overall may still contain a garbage footer line worth stripping.

Markdown emission after HTML clean should preserve lists and tables that survived DOM strip. Flattening lists into paragraphs during clean undoes structure the chunker needs. Prefer structure-preserving cleaners when the downstream splitter is heading-aware.

Unicode control characters and soft hyphens sometimes remain after NFKC. Add a small post-normalisation pass that removes soft hyphens and zero-width characters that fragment tokens in BM25 and in embeddings.

Language detection after clean can feed metadata language fields. Running detection before clean biases toward chrome language in multilingual footers. Detect after strip when footers are multilingual spam.

Security note: cleaning is not redaction. Secrets in body text must be handled by dedicated secret scanning before embed. Strip rules that remove “password” labels but leave values are unsafe. Run secret scanners as a separate gate.

PII in footers such as support phone numbers may be intentional contact data or unintentional leakage depending on product scope. Decide keep or strip in the cleaning profile and document it for privacy review.

Batch re-clean jobs should be idempotent: cleaning already-cleaned text should not delete more content. Write rules against stable markers or store a cleaned flag. Non-idempotent rules cause gradual erosion across reprocessing.

When using managed cleaners, export both cleaned text and a machine-readable list of removed categories if the vendor provides it. Without categories, you cannot tell ads from footers in incident review.

Open-source stacks that combine Trafilatura with Unstructured or Docling outputs should clean after the parser that produced the string. Cleaning Unstructured NarrativeText differently from HTML main content is normal; do not force one regex set on both without tests.

Regex strip lists for “All rights reserved” and similar are useful supplements to frequency heuristics for rare but toxic phrases. Keep regex lists short and reviewed; sprawling regexes become unmaintainable and over-match.

Human-in-the-loop review queues should sample both high-deletion and low-deletion outliers. High deletion may be over-strip. Low deletion on known-noisy sites may be under-strip. Mid-distribution samples teach less per reviewer minute.

Chunking overlap settings cannot compensate for chrome. Overlap will duplicate footers into more chunks and worsen dilution. Clean first.

Embedding model upgrades also cannot compensate for chrome. A stronger model still embeds menu tokens if menus remain. Spend upgrade budget after clean gates pass.

Finally, align cleaning profiles with connector sources: one profile for Notion HTML, one for SharePoint web parts, one for scanned PDF OCR text, one for digital PDF parse text. Shared rulesets that ignore source family underperform.

Site search boxes and “type to search” placeholders sometimes appear in extracts as body text. They retrieve for queries about search itself and confuse users. Strip search chrome explicitly in HTML profiles.

Social share button labels (“Share on LinkedIn”) are classic false lexical hits for brand queries. DOM strip of share widgets removes them before frequency heuristics are needed.

Print-only CSS content that appears in some extractors should be evaluated: sometimes print headers are useful titles; sometimes they are repeated chrome. Prefer visibility heuristics that match what a reader sees on screen for RAG corpora meant to answer screen readers’ questions.

Collapsed accordion help text that is hidden in the DOM but present in HTML source will be extracted by naive scrapers. Decide whether hidden FAQ accordion content is in scope. If yes, keep; if the product shows it only on click and your corpus policy excludes it, strip. Document the choice.

Duplicate body copies from “amp” or printer-friendly sibling pages should be handled by corpus dedup rather than by line strip. Cleaning inside a document will not merge two near-duplicate URLs. Route that to preparing your data for RAG or deduplicating context as appropriate.

Table of contents blocks that mirror headings are often strip because the headings remain in the body. Keeping both doubles heading tokens. Prefer strip TOC when headings are preserved; keep TOC only when body headings were lost at parse.

Author bio boxes at the end of articles may be keep for “who wrote this” questions and strip for product FAQ bots. Profile again. Default for product support corpora is strip; default for research corpora is keep.

Related-article carousels inject foreign titles into the extract. Those titles retrieve without the related article body. Strip carousels; if related content matters, ingest those URLs as separate documents via connectors.

Timestamp lines like “Last updated” are often keep because freshness questions need them, and they also feed metadata. Prefer promoting them into metadata fields and then stripping the raw line if it would otherwise duplicate.

Category tags and topic chips in blogs are borderline: they help topic filters if promoted to metadata, and they pollute embeddings if left as body. Promote then strip is the clean pattern.

Footnote section headers labelled “References” should be kept when citations are answers; the reference lines themselves may be keep or strip depending on whether users ask for sources. Academic assistants usually keep references; consumer assistants often strip.

When measuring token delta, use the same tokenizer the embedding model uses, not a whitespace split. Whitespace deltas understate BPE token waste from chrome punctuation.

Report cleaning results beside parser strategy in ops dashboards so a spike in strip volume after a parser change is attributable. Parser upgrades that suddenly inline nav into body will look like cleaning regressions if dashboards are siloed.

Training data for LLM strippers should include keep examples of operative clauses that resemble disclaimers. Without those positives, models over-strip legal text. Review training mixes the same way you review strip regexes.

Exit criteria for this leaf in a production readiness review: fixtures green in CI, token delta measured on a held-out sample, negative chrome queries failing closed, operative-clause keeps verified by a human on YMYL samples, and cleaning_version wired into the incremental registry. Until those pass, do not scale embed.

Mobile app webviews that wrap help articles often inject native navigation labels into HTML snapshots used for ingest. Capture pipelines should prefer canonical web article URLs over webview HTML when both exist, then apply the HTML cleaning profile once.

PDF portfolio containers and attachments require separate extract passes; cleaning the cover PDF does not clean attached memos. Enumerate attachments at parse time, clean each child document, and preserve parent-child metadata so citations remain navigable.

When strip rules disagree across teams, escalate to a written cleaning profile owned by the product, not to ad hoc regexes in personal branches. Unowned rulesets drift and create unreproducible indexes.

Seasonal promo banners that change weekly will defeat static regex lists. Prefer DOM selectors on banner containers or short-lived frequency windows that recompute chrome candidates nightly for marketing sites with rotating campaigns.

After cleaning passes the gate, re-run a small golden retrieval set before declaring the index healthy. Cleaning changes rankings; confirm that intended answer chunks still surface for the questions you already trust in evaluation and staging checks before full re-embed of the affected corpus slice only, nothing more.

Document cleaning for RAG is finished for a batch when sample pages show lower token counts without losing operative clauses, navigation-only queries stop retrieving chrome, OCR gibberish lines are gone, and cleaned Markdown or text is ready for the chunker. Everything after that is chunking and embedding. Everything before that is still a parse string pretending that menus are knowledge. Strip chrome, keep answers, measure the delta on your own sources, and refuse vendor percentages as law until they survive your fixtures.

Across ChatNexus, SearchCans, Drive AI, Diomgis, bthek1, Databricks cleaning notes, Microsoft Learn preparation material, Weaviate pipeline writing, and mbrenndoerfer-scale guides filtered for cleaning-relevant claims, the shared demand is the same: remove chrome before split, and do not confuse cleaning with chunking strategy. Strip-versus-keep keyed to retrieval, three strategy families with explicit failure modes, and inheritance framing after clean are the mechanism claims that removal-only checklists omit.

HTML-heavy corpora should instrument Trafilatura or Readability outputs beside raw DOM dumps for a week and compare false-hit rates. PDF-heavy corpora should instrument fixed-position strip beside layout-aware parse upgrades; often the cheaper win is better parse, not cleverer strip. Mixed corpora need both.

Teams sometimes try to “prompt away” chrome by telling the generator to ignore headers. That fails when retrieval never surfaces the body because chrome chunks ranked higher. Cleaning is index-time; prompt instructions are not a substitute for deleting nav tokens from the store.

Another failed substitute is lowering top-k until chrome disappears. That reduces recall on real answers that share lexical tokens with menus. Strip the menus.

International sites need language-aware chrome lists: “Accueil” and “Kontakt” are navigation too. Maintain per-locale strip dictionaries or rely on DOM structure that does not care about the label language.

Authenticated app help centers often duplicate the same left nav into every article extract. Frequency heuristics excel there. Public marketing blogs with unique promo blocks per article may need DOM rules more than frequency.

When cleaning and metadata interact, remember the order: parse, clean, enrich. Enrichment that tags chrome which should have been deleted wastes enrichment budget and can mark junk as “policy” if LLM classifiers see confidentiality footers.

Evaluation should include negative questions that should not be answered from chrome, such as asking for a phone number that appears only in a footer sitewide. After cleaning, those questions should miss rather than return the footer. That negative test is as important as positive answer tests.

Storage of pre-clean artifacts enables audits when a regulated user asks why a clause disappeared. Keep raw and cleaned for a retention window that matches your compliance policy. Deleting raw immediately saves storage and destroys forensics.

Finally, treat cleaning quality as a release gate equal to parser quality. A model upgrade will not repair an index full of cookie banners. Re-clean and re-embed when strip rules change materially, using incremental registries keyed by content hash after clean, not before.

More in Document Ingestion for RAG Toggle section list