Chunking by Document Structure: Markdown, HTML and Headings

Structure-aware chunking splits a document on its own hierarchy — Markdown and HTML headings, sections, and other authored boundaries — instead of every N tokens. Each chunk is meant to be one coherent section, usually carrying the heading path as metadata so retrieval knows where it sat. The method wins when that hierarchy is real; it fails when the structure is a lie. This page covers how the split works for Markdown and HTML, why the heading path matters, what to do when headings are fake, and when recursive character chunking is the safer fallback. The full strategy map lives on the chunking hub.

How does structure-aware chunking decide where to split?

Structure-aware chunking turns the document into a typed tree, then cuts where authors already marked topic changes — not where a token counter happens to land.

1. Parse into typed elements. Headings (with levels), paragraphs, lists, code fences, and tables become nodes with hierarchy. Markdown exposes that tree explicitly; HTML exposes it through the DOM; layout models (for example Azure’s Document Layout skill on Microsoft Learn, as of July 2026) infer heading-like structure from PDFs and office files.
2. Group under headings. Walk the tree and keep content that shares a heading path in the same candidate chunk. Atomic units stay intact: do not bisect a fenced code block or a table mid-row — those failures belong to code chunking and table chunking.
3. Size-normalize inside sections. Oversized sections split on sub-headings first, then with a recursive character splitter within the section. Undersized stubs merge only with neighbours under the same heading — never across a major heading transition (Samuel Ochoa, structure-aware chunking guide, updated April 2026; Hariprasannaa, Medium, December 2025).

Recursive character chunking is related but not the same mechanism: it tries a separator hierarchy (nn → n → spaces → characters) without building a heading tree or heading-path metadata. In strong pipelines recursive is the component you run after structure when a single section still exceeds the size budget.

How does Markdown header chunking work?

Markdown header chunking splits on declared heading markers and attaches the heading path to each chunk’s metadata. LangChain’s MarkdownHeaderTextSplitter in langchain-text-splitters (docs.langchain.com, live July 2026) is the reference pattern most tutorials copy.

You declare which levels count as boundaries — typically (“#”, “Header 1”), (“##”, “Header 2”), (“###”, “Header 3”) — and call split_text. For a document that opens # Foo then ## Bar with two lines of body, LangChain’s worked example emits one document whose metadata is {‘Header 1’: ‘Foo’, ‘Header 2’: ‘Bar’} and whose content is the Bar body. By default the splitter strips the header lines from page_content; set strip_headers=False to keep them in the text as well as the metadata.

Production pipelines almost never stop there. Sections vary wildly in length, so the documented hybrid is header-split first, then RecursiveCharacterTextSplitter.split_documents on the oversized sections (LangChain docs “How to constrain chunk size”; MDisBetter, May 2026; Dataknobs markdown chunking; Oleh Halytskyi, DEV, header-then-token pattern). Use split_documents, not a fresh split_text on the joined corpus: LangChain’s troubleshooting notes that overlap applies inside a section and does not cross H1/H2 boundaries. LlamaIndex’s MarkdownNodeParser follows the same section-as-unit idea (Ochoa). Chunk-size and overlap knobs after the header pass belong on chunk size and overlap — not invented “best” defaults.

How does HTML chunking work for RAG?

HTML structure chunking parses the DOM and splits on heading tags or section containers, then keeps the heading path the same way Markdown does — but the input is messier.

LangChain ships HTMLHeaderTextSplitter, HTMLSectionSplitter, and HTMLSemanticPreservingSplitter in the same langchain-text-splitters package (API index, live July 2026). LlamaIndex exposes HTMLNodeParser. IBM’s RAG cookbook (Think / architectures) frames the practical moves: parse on tags such as <p> and <div>, extract main text and headings, and treat each meaningful page section as a candidate unit. Two workable patterns dominate: split headers in place on the HTML, or convert clean HTML to Markdown and reuse the Markdown header splitter (Ochoa’s “HTML case”; Firecrawl’s scrape-to-markdown framing for web corpora).

Older or scraped HTML is where structure-aware methods quietly degrade: non-standard tags, nested deprecated markup, and inline style chrome produce fragmented or nav-heavy chunks (IBM, challenges with older HTML). For PDFs and office files without a native heading tree, Microsoft’s Document Layout skill (Azure AI Search, Microsoft Learn) detects layout headings into fields such as header_1…header_3, then a Text Split skill constrains chunk size inside each Markdown-like section — structure inferred, not authored.

Why should each chunk carry the heading path?

Every structure-aware chunk should carry its heading path — in metadata, prepended text, or both — so the embedding and the generator know which section the passage belongs to.

Ochoa’s heading-context example is the reason: a chunk under Setup > Authentication > OAuth > Configuration is no longer an ambiguous “rate limits” fragment that could mean API limits, OAuth token limits, or webhook limits in the same manual. Unrag’s structure-aware guide makes the same move — prepend parent headings so a subsection chunk is self-describing. Dataknobs stores the full root-to-chunk heading list on chunk metadata; LangChain stores Header 1 / Header 2 / Header 3 keys on each Document. Without the path, repeated titles such as “Overview” or “Limitations” collide across documents and sections. This is not a separate “metadata strategy”; it is part of structure-aware chunking. When those heading-defined sections become parents for small-to-big retrieval, continue on hierarchical and parent-document chunking.

What do you do when the document structure is a lie?

Structure-aware chunking trusts author boundaries. When those boundaries are fake, it amplifies noise instead of meaning — so validate the hierarchy before you commit the index to it.

• Decorative or SEO headings that do not mark topic changes create false section cuts.
• Nav, chrome, and sidebar headings parsed as H2s pull menus into the retrieval index (IBM’s older-HTML failure modes; messy web scrapes).
• PDF→Markdown or OCR pipelines invent heading levels from font size and position; chunk quality then tracks parser quality, not author intent (Ochoa, PDF case).
• Stub headings with a single thin sentence produce weak embeddings and low recall (Hariprasannaa on uneven section sizes).
• Missing headings make a header splitter return one giant blob or near-useless fragments (MDisBetter’s stated cons for header-only splitting).

When should you use structure-aware chunking instead of recursive?

Use structure-aware chunking when documents have a real heading hierarchy — technical docs, wikis, manuals, and Markdown/HTML knowledge bases — where section boundaries are the semantic units you want to retrieve (Ochoa; Unrag; Azure Document Layout guidance; MDisBetter’s “right default for Markdown”). Skip it or treat it as overkill on plain prose blogs without H2s, chat logs, emails, and other homogeneous short posts (Ochoa “when it’s overkill”; Unrag on flowing prose).

Prefer recursive character chunking as the generic fallback when the text has paragraph separators but no trustworthy tree. Cost class matches recursive, not semantic: structure-aware splitting uses string or DOM parsing only — zero embedding or LLM calls at split time — unlike semantic chunking, which embeds every sentence to find breakpoints. After real section parents exist and answers still span more than one chunk, escalate to hierarchical / parent-document chunking. Prove the choice on your own chunking evaluation; do not cite unpublished Top-1 percentage tables as laws.

How do you implement structure-aware chunking?

Frameworks already ship the splitters; the mechanism page names the pattern, not a full notebook. In LangChain (as of July 2026) install langchain-text-splitters, run MarkdownHeaderTextSplitter (or HTMLHeaderTextSplitter for HTML), then RecursiveCharacterTextSplitter.split_documents on oversized sections. LlamaIndex’s path is MarkdownNodeParser / HTMLNodeParser. Azure pipelines can use the Document Layout skill plus Text Split for layout-detected headers (Microsoft Learn). Pin library versions, show outputs, and wire the runnable end-to-end flow on building the pipeline; measure retrieval against recursive and fixed baselines on chunking evaluation; tune length after headers on chunk size and overlap.