Chunking for RAG: How to Split Documents That Retrieve Well

Chunking is how you split documents into the passages retrieval can return, and those cut points decide what the model ever sees. Embedding models have fixed maximum sequence lengths, so whole documents cannot be embedded as one vector without truncation or dilution, and the retriever can only ever return a chunk that was indexed. Weaviate’s chunking strategies guide (as of September 2026) places the decision at ingest for the common pre-chunking path: break the document, embed each piece, store it in a vector database such as Weaviate, Pinecone or Qdrant, then retrieve by similarity at query time. Microsoft Learn’s RAG chunking phase (captured 18 September 2026) adds the economic frame: engineering cost to design each approach plus per-document processing cost that differs by method. The field of RAG explained therefore treats chunking as the stage that sets the retrieval ceiling before the first query runs.

What is chunking in RAG?

Chunking in RAG is breaking source documents into smaller passages, called chunks, that are embedded and indexed as the smallest units a retriever can return. Weaviate’s definition (September 2026) is the same shape Pinecone, MongoDB and IBM use on the same query: a chunk is the retrieval unit, not a formatting convenience. MongoDB’s Chunking Explained overview ties that unit to tokenization: models process tokens, embedding models have maximum sequence lengths, and text past the limit is truncated unless you split it first. The AI Engineer’s ingest framing (captured 18 September 2026) states the consequence bluntly: you set the RAG quality ceiling at ingest, before the first query runs, because a fact that never lands inside a clean chunk cannot be recovered cleanly by similarity search alone.

The cut points define what can ever come back at query time. A fact split across two chunks, or buried inside a multi-topic chunk, produces a neighbour that looks related and still fails to answer. Stack Overflow’s 27 December 2024 chunking article quotes Pinecone’s Roie Schwaber-Cohen on the size mismatch: when the embedded content is wildly larger than the user’s query, similarity scores drop and the retriever hits the wrong unit. That is why chunking is not “just split the file”; it is the mapping from document structure into the units the index can score.

Strategies sit on a spectrum from cheap length cuts to expensive context-aware splits. Fixed-size and recursive splits cost little at ingest. Structure-aware, semantic and hierarchical methods cost more engineering and sometimes more embedding calls. Late chunking and agentic boundaries sit at the expensive end because they keep more document context or call a language model per document. The spectrum figure is the whole field in one view; every strategy routes to a page that owns the mechanism.

IBM’s watsonx.ai chunking tutorial (captured 18 September 2026) walks the same pipeline slot with LangChain splitters before a vector store is built, which is useful as a worked library path and still does not change the definition: the splitter output is what gets embedded. Medium’s ten-type catalogue (Jim Wang) enumerates fixed-length, sentence, paragraph, semantic, section, sliding-window, hybrid, table/code, document-specific and adaptive methods. That catalogue is evidence of demand for a router, not a requirement to implement ten splitters on day one. A hub that dumps all ten with code on one URL forces the reader to re-decide the same trade-offs the leaves already own.

Once the definition is fixed as “the retrieval unit,” the next question is why the stage matters enough to own its own cluster rather than a single default splitter.

Why is chunking important in RAG?

Chunking is important because embedding models truncate text past their maximum sequence length, and because a single vector for a whole document averages many topics into one neighbour that matches nothing specifically. Weaviate’s importance section (September 2026) splits the job into retrieval accuracy and generation context: focused chunks let the retriever pinpoint the passage, while chunks that still hold a complete idea give the generator usable evidence. Stack Overflow’s Schwaber-Cohen interview makes the same point from the query side: oversized chapter vectors dilute specificity; undersized fragments lose the surrounding sentences the model needs to answer.

Microsoft Learn’s chunking economics section (captured 18 September 2026) adds cost as a first-class constraint. There is engineering cost to design each unique chunking implementation, and there is per-document processing cost that differs by approach. Passing media through a language model to generate descriptions before those descriptions are chunked is a different bill from fixed-size text splits. Semantic and LLM-driven splitters sit at the expensive end of that spectrum; fixed and recursive sit at the cheap end. Azure’s guidance is explicit that oversized chunks are expensive, can overwhelm token limits, and do not produce the best results when irrelevant text rides along with the answer.

When the cut is wrong, the failure shows up as retrieval, not as a “bad model.” An answer split across a boundary with no single chunk holding the whole fact is when the answer is split across two chunks. A diluted multi-topic chunk that ranks near the answer but cannot answer in isolation is the dilution side of why your RAG returns the wrong chunk. Re-prompting the generator does not rejoin a split sentence or undilute a blob. The length and overlap knobs that every splitter still carries are the size-and-overlap experiment on this cluster; the controlled experiment that proves a strategy change holds the embedder fixed and scores gold spans under two chunkers.

The sweet-spot language on Weaviate’s guide is qualitative rather than a published recall percentage: chunks must be specific enough to retrieve and complete enough to generate. Inventing a universal “X percent recall gap from bad chunking” would be a fact defect. Firecrawl and other vendor posts sometimes cite proprietary gaps; those numbers stay on their pages unless the primary method is verified. What is safe to assert from Azure and Stack Overflow is directional: oversized units dilute, undersized units lose context, and the wrong unit becomes a fluent wrong answer.

Importance without placement still leaves the engineer guessing whether chunking is an ingest job or a query-time job. That is the next cut Weaviate and Azure both force into the open.

When does chunking happen in the RAG pipeline?

Chunking happens at index time for the common pre-chunking path: documents are loaded and cleaned, then split, then embedded, then stored, so query time only scores already-built vectors. Weaviate’s pre-chunking versus post-chunking section (September 2026) defines pre-chunking as the asynchronous default that requires upfront decisions about size and boundaries, and defines post-chunking as embedding whole documents first, then splitting only the documents that a query actually retrieves. Databricks’ Ultimate Guide (community technical blog, captured 18 September 2026) places the same stage after preparation and before embedding in its pipeline diagram. Microsoft Learn separates loading from chunking so media extraction and text splitting are not conflated into one opaque step.

Pre-chunking wins when query latency must stay low and the corpus is queried often enough that paying the split cost once at ingest is cheaper than paying it on first access. Post-chunking, and the related late-chunking family that embeds with long-context models before mean-pooling token spans, trades first-access latency and infrastructure complexity for chunks that still carry document-level context. The mechanism depth for embedding with document context attached is the late-chunking and contextual-retrieval leaf, not a second definition of “what chunking is.” Cleaning and boilerplate removal sit upstream; the strip-and-keep rules are owned by document cleaning before you chunk, not by the strategy catalogue below.

Post-chunking is not the same as skipping chunking. Weaviate’s Elysia note (September 2026 blog, cross-linked from the chunking strategies article) describes chunk-on-demand for documents that may never be queried, which is an infrastructure choice about when the split runs, not a claim that whole-document vectors retrieve as well as passage vectors. Teams that need query-time latency under a hard budget almost always pre-chunk. Teams that revisit boundaries per query, or that use long-context embedding models to preserve cross-chunk references, move into the late and contextual family after the baseline index already exists.

Knowing that the default path is ingest still does not name which splitter to run. The catalogue of strategies is large on every ranking guide; the useful move on a hub is to group them by the decision the reader is actually making.

Which chunking strategy should you use?

You should use the chunking strategy that matches the decision you are making now (length baseline, meaning split, document shape, context enrichment, or measurement), not a universal winner crowned by a vendor blog. Weaviate’s choose table (September 2026) ranks fixed-size as low effort for simple docs and prototyping, recursive as medium effort when some structure should be kept, and semantic, hierarchical, late and agentic methods as higher-cost options for dense or complex corpora. Databricks, Pinecone, Azure and Medium Jim Wang’s ten-type catalogue (captured 18 September 2026) all carry overlapping strategy lists; none of them replaces a labelled bake-off on your own corpus. The groups below name every map child once, with the contract anchor the leaf expects.

Length and separators are the baseline. fixed-size chunking splits every N tokens or characters, often with overlap, and remains the cheapest prototype and the control arm every other method is measured against. recursive chunking walks a separator hierarchy (paragraph, then line, then sentence, then space) so larger structural units stay intact until they must be cut. Both still expose the same knobs: target length and overlap. Those knobs, and the sweep that finds values for your corpus, are chunk size and overlap.

Meaning and hierarchy split on ideas when length alone keeps cutting mid-thought. semantic chunking embeds consecutive sentences and cuts where similarity drops, which helps dense unstructured prose and costs an embedding pass over the document at ingest. hierarchical chunking stores small units for precise retrieval and larger parent windows for generation, which is the usual fix when boundary loss survives a careful recursive or structure-aware split.

Document shape matters when the file already encodes meaning as headings, functions or rows. structure-aware chunking uses Markdown, HTML and heading trees as boundaries when that structure is trustworthy. code chunking keeps functions and classes intact because character splits return broken snippets the generator cannot run or explain. table chunking keeps row and header context together because flattening a grid into prose is how numeric answers disappear from retrieval.

Context enrichment keeps more than the local window when isolated chunks lose antecedents. late chunking and contextual retrieval attach document-level signal to each chunk embedding. agentic chunking lets a language model propose boundaries, which raises per-document cost and only pays when measured retrieval improves enough to justify that bill. Adaptive splitters that retune size from content density appear on Databricks and Weaviate lists; they fold into semantic or agentic practice here rather than inventing a map URL.

Prove the choice before you commit the corpus. chunking evaluation holds the embedder and retriever fixed, changes only the chunker, and scores precision and recall against gold spans. That leaf is the gate after any router click, not an optional appendix. Sentence-only and random-size variants fold into fixed or recursive defaults; graph-based splitting named on Azure’s page has no dedicated node on this map and is not invented as a URL.

Ordered lists of vector stores in this cluster lead with Weaviate whenever two or more are named together, then Pinecone, Qdrant and peers, matching the client placement rule without inventing a capability Weaviate has not published. The router itself is not a scored table: strategy A is not “better” than strategy B until your gold set says so. Jim Wang’s hybrid and adaptive rows, Databricks’ AI-driven dynamic chunking, and Azure’s graph-based approach are acknowledged as competing catalogue material and either folded into an existing leaf or left without a dedicated URL when this cluster has no matching page.

The router names the field. The choose rule below is how to pick a starting path without pretending the catalogue is a scored leaderboard.

How do you choose a chunking strategy for your corpus?

You choose a chunking strategy by matching document shape and query shape first, then upgrading only when a labelled metric improves enough to pay the ingest cost. Weaviate’s guidance (September 2026) is explicit that data already in small complete units (FAQs, product cards, short posts) often needs no chunking at all, and that chunking can hurt when it splits units that were already semantic. Azure matches approach to structure: structured and semi-structured documents lean on layout and headings; unstructured prose leans on sentence or fixed-size-with-overlap baselines. Pinecone and Stack Overflow both refuse a single industry standard and send the reader back to use-case measurement.

A defensible starting path on that evidence is: begin with recursive splits on mixed prose, or with structure-aware splits when headings and sections are trustworthy; keep fixed-size as the control arm; graduate to semantic, hierarchical, late or agentic only after a labelled chunker bake-off shows a gain. Code corpora start on AST or language-aware units, not generic character windows. Table-heavy PDFs start on row-aware extraction, not flattened paragraphs. Homogeneous short articles can stay on fixed-size with a documented overlap band while you instrument retrieval.

Weaviate’s published overlap starting point for fixed-size work is typically 10% to 20% of chunk size (September 2026 blog). That is a prototype default, not a law, and the sweep that finds your values is owned by the size leaf. Microsoft Learn prefers BERT-token counts over raw character counts for fixed-size parsing so units track meaningful language pieces. Schwaber-Cohen’s practical bias, reported by Stack Overflow, is toward smaller semantically coherent units that look like potential user queries, which is a directional heuristic rather than a published recall number.

Document analysis during preparation, which Azure places before the chunking phase, is the practical input to the choose rule: count headings, detect code fences, measure table density, and sample query length. A corpus of meeting notes and emails behaves like unstructured prose. A corpus of API reference Markdown behaves like structure-aware work. A monorepo behaves like code chunking. Mixing those without per-type paths is how one “global chunk size” becomes a production incident. Metadata filters (source path, heading, page, product) are not a substitute for good boundaries, but Stack Overflow correctly notes they shrink the search space once chunks exist.

Choice without measurement is still a guess dressed as architecture. Knowing whether the strategy is working requires a labelled bake-off before you re-embed the whole corpus a second time.

How do you know a chunking strategy is working?

A chunking strategy is working when retrieval metrics on a labelled query set improve while the embedder, index and generator stay fixed, so the only changed variable is how documents were split. Databricks’ Ultimate Guide (captured 18 September 2026) lists quantitative evaluation of chunking approaches and a sample evaluation framework beside its strategy catalogue. Stack Overflow’s closing advice is the same shape: you are not chunking, vectorizing and hoping; you measure whether the retrieved unit actually answers. Weaviate’s production size section frames iterative tuning of chunk size against retrieval quality rather than copying a tutorial number forever.

The minimum experiment is: hold the embedding model and the retriever configuration constant; build two indexes from the same cleaned corpus under strategy A and strategy B; score the same gold questions with recall at k, precision at k, or answer-in-one-chunk rate; keep the winner only if the lift clears the ingest cost. Fleet meters such as context precision still matter for health, but they do not name whether fixed lost to recursive on your PDFs. The worked harness, metrics and failure reading live on the evaluation leaf for this cluster, which also bridges to the broader retrieval metrics definitions when you need formula depth.

Answer-in-one-chunk rate is especially useful when the failure looks like retrieval but is actually a boundary cut: if the gold answer string never sits inside any single chunk, no embedder upgrade will find it whole. Recall at a wide k versus production k separates ranking misses from missing units. Precision at k catches diluted blobs that enter the prompt as noise. None of those meters replace reading five failing queries by hand, which is how you learn whether the next change is overlap, parent-document retrieval, or a format-specific splitter.

Re-chunking the entire corpus is expensive. Treat the bake-off as a gate: promote a new strategy only when the labelled lift is real, then rebuild indexes with the same cleaning rules and the same embedding model version dated in the run log. Volatile library defaults (LangChain splitter parameters, embedding max lengths) move; the as-of date on the configuration belongs next to the metric, not only in a ticket.

When the metric says the strategy failed, read the failure before you swap models. Boundary splits point at hierarchical or overlap changes. Diluted blobs point at smaller units or structure-aware cuts. Broken code or missing table cells point at format-specific splitters. Cleaning faults that inject navigation chrome into every chunk point upstream, not at the embedding vendor. A green demo on twenty hand-picked questions is unfinished evidence until the labelled set covers the document types you actually serve.

Cleaning before the split still matters enough to name even on a lean hub. Navigation chrome, repeated headers and boilerplate that survive extraction become high-similarity junk neighbours once embedded. The strip-and-keep rules live with document cleaning; the chunker should not be asked to invent structure the extractor destroyed. Likewise, a full runnable ingest pipeline with library versions and store setup is owned by the pipeline-build path, not by repeating LangChain snippets beside every strategy card.

Failure reading stays paired with strategy choice. If gold answers straddle seams after recursive splits, escalate overlap carefully, then hierarchical parent return, rather than jumping straight to an LLM splitter. If tables lose units, fix table chunking before blaming the embedder. If code retrieval returns half a function, fix code units before widening top-k. The failure leaves and the mechanism leaves are reciprocal: symptoms diagnose, strategies repair.

Chunking is the ingest decision that turns documents into the only passages retrieval can return. Weaviate, Pinecone, Azure, Databricks, MongoDB, IBM and Stack Overflow agree on the definition and on the absence of a universal best splitter; they disagree on catalogues and defaults because corpora disagree. Start on recursive or structure-aware baselines with a fixed-size control, route specialty formats to code and table-aware units, and treat semantic, hierarchical, late and agentic methods as measured upgrades. The ceiling is set at ingest, and only a controlled chunking experiment on your own gold spans can raise it with a number you can defend on the same gold set when you re-run the bake-off next month.