Semantic Chunking: Splitting on Meaning Instead of Length
Embedding-distance splitting measured against fixed-size on the same corpus — including the cases where it loses.
Semantic chunking splits a document at the points where its meaning shifts, instead of every N tokens. It embeds each sentence, measures how similar neighbouring sentences are, and cuts where that similarity drops. The promise is chunks that hold one coherent idea each; the price is an embedding call for every sentence at ingest. This page covers how it works, what that costs, and the corpora where it actually beats plain fixed-size chunking — and the ones where it doesn’t.
How does semantic chunking decide where to split?
The mechanism is three steps, run once at ingest time:
- Split into sentences. The document is broken into candidate units — usually sentences, sometimes small groups of them.
- Embed each unit. Every sentence is turned into a vector with the same embedding model you retrieve with.
- Cut at the similarity drops. Walk the sentences in order and compare each to the next by cosine similarity. Where similarity falls below a threshold — a breakpoint — start a new chunk. A run of sentences about the same thing stays together; a topic change forces a boundary.
The one knob is that threshold. Set it strict and you get many small, tightly-focused chunks; set it loose and you get fewer, broader ones. Most implementations express it as a percentile of the observed similarity gaps rather than an absolute number, so it adapts to the document.
What does semantic chunking cost?
This is the half most guides skip or wave at. Semantic chunking is not free, and the cost is structural, not incidental:
- One embedding call per sentence, at ingest. A fixed-size splitter makes zero model calls to decide a boundary — it counts tokens. Semantic chunking embeds every sentence to find the boundaries, so a 10,000-sentence corpus is 10,000 extra embedding calls before a single query is ever run.
- Ingest latency and spend scale with document count. Those calls are money (per-token embedding pricing) and time (added to your indexing pipeline). Microsoft’s Azure RAG guidance is blunt that chunking has real “economics” — per-document processing costs that differ by approach — and semantic chunking sits at the expensive end.
- It is an index-time cost, not a query-time one. The good news: you pay it once per document, not per query. Re-embedding only happens when the document changes.
Measure it on your own corpus before committing
The exact latency and dollar delta versus fixed-size depend on your embedding model’s price, your average document length, and your ingest volume — so put a real number on it for your pipeline rather than trusting a blog’s. The structural fact holds everywhere: one embedding call per sentence versus none.
Is semantic chunking better than fixed-size?
Semantic chunking earns its cost when the document’s topics are uneven — where a fixed-size cut would slice through the middle of an idea:
- It wins on heterogeneous documents — mixed-topic pages, FAQs, transcripts, docs that jump between subjects. Keeping each idea whole is exactly where retrieval accuracy improves.
- It barely helps on uniform text — a single-topic article of even paragraphs chunks almost identically whether you split on meaning or on length, so the extra embedding cost buys little.
- It can hurt on very short or very structured content — code, tables, and lists have their own natural boundaries that sentence-similarity misreads; those want structure-aware splitting instead.
The honest rule: reach for semantic chunking when your corpus is messy and mixed, and don’t pay for it when your text is already uniform. Prove which case you’re in by measuring retrieval on the same corpus, both ways — the benchmark, not the blog, decides.
What failure does semantic chunking prevent?
Semantic chunking exists mainly to prevent one failure: an answer cut across a chunk boundary, where no single retrieved chunk contains the whole answer. It reduces that, but it is not immune — a threshold set wrong still splits mid-idea, and a fact that genuinely spans a real topic change can still land in two chunks. Chunking changes the odds; it does not remove the failure mode.
How do you implement semantic chunking?
Every major framework ships a semantic splitter (LangChain, LlamaIndex, and others), and the difference between them is mostly how they pick the breakpoint. Rather than reproduce a full walkthrough here — that belongs with the runnable pipeline — see building the pipeline for the minimal, pinned, output-shown version, and the chunking benchmark for which splitter actually retrieves best on measured data.
What is semantic chunking?
Semantic chunking splits a document where its meaning changes rather than every fixed number of tokens. It embeds each sentence, compares neighbouring sentences by similarity, and starts a new chunk wherever similarity drops — so each chunk holds one coherent idea.
Is semantic chunking worth the extra cost?
It depends on your documents. On heterogeneous, mixed-topic content it improves retrieval enough to justify the one-embedding-call-per-sentence ingest cost. On uniform single-topic text it chunks almost identically to fixed-size, so you pay for little gain. Measure both ways on your own corpus before committing.
How do I set the similarity threshold?
Most implementations express the breakpoint as a percentile of the observed similarity gaps rather than an absolute number, so it adapts per document. Start with the framework default, then tighten it if chunks are too broad or loosen it if they fragment mid-idea.
When is fixed-size chunking good enough?
When your text is uniform and single-topic, fixed-size with sensible overlap chunks almost identically to semantic — at zero extra embedding cost. Fixed-size is also the right baseline to benchmark against before adopting anything more expensive.
Does semantic chunking prevent chunk-boundary failures?
It reduces them but does not eliminate them. By cutting at topic changes it keeps most answers whole, but a mis-set threshold or a fact that truly spans a topic shift can still land across two chunks. It changes the odds, not the failure mode.