Chunk Size and Overlap: How to Choose Them

Chunk size is the token (or character) length of each retrieval unit you embed; chunk overlap is the shared seam of tokens repeated between neighbouring chunks. There is no published universal optimum — only starting bands and a measurable trade-off between precision and context on your corpus. This page covers that trade-off, what overlap actually buys, how to pick a starting point, and the sweep that finds yours. The full strategy catalogue lives at chunking.

How does chunk size affect retrieval quality?

Chunk size sets how much text collapses into a single embedding vector. Smaller windows favour precise matches; larger windows favour complete context — and every length pays the other side’s cost:

• Smaller chunks (about 128–256 tokens) match factoid and lookup queries well because the vector is dominated by one idea. The risk is missing surrounding context — pronouns, exceptions, and prior sentences that a human needs to interpret the hit (vinish.dev; Stackviv 2026).
• Larger chunks (about 512–1024+ tokens) give the generator enough passage to answer analytical questions. The risk is dilution: one vector averages several topics, so a specific query can miss the chunk or retrieve it with heavy noise (Ertas 2026).

Embedding models emit one fixed-size vector per chunk regardless of input length, so length is always a density-versus-coverage trade. Chroma’s 2024 chunking evaluation made the dilution visible at token level: on their text-embedding-3-large table, recursive splitting at 200 tokens with 0 overlap reached about 7.0% mean token-level precision, while recursive at 800 tokens with 400-token overlap fell to about 1.5% — a published measurement on that setup, not a universal law. Prove the trade-off on your labelled set with a controlled chunking experiment.

What does chunk overlap do?

Chunk overlap repeats the last N tokens (or a percentage) of chunk i as the start of chunk i+1, so a short fact that straddles a seam can land wholly inside at least one chunk. Without overlap, a sentence that starts near the end of one window and finishes in the next is split across two embeddings — neither of which holds the full meaning (Ertas 2026; ByteTools 2026).

The cost is structural: more overlap means more chunks per document, which means more embeddings to store and search. Microsoft’s Azure AI Search guidance shows the effect on NASA’s Earth at Night e-book with the Text Split skill — same source, different overlap:

Zero overlap is only appropriate when boundary detection already keeps meaningful units whole — typically section-level splits on well-structured documents (Ertas 2026). When the answer is cut across a seam, diagnose it on chunk-boundary loss before you raise overlap blindly.

How much chunk overlap should you use?

There is no universal best overlap percentage. Published starting bands cluster around 10–20% of chunk size (Ertas 2026; CustomGPT guidance; Bswen’s worked example uses 200 overlap on a 1000-unit chunk ≈ 20%; ByteTools 2026). Microsoft’s Azure guidance recommends starting at 512 tokens with about 25% overlap (≈128 tokens) — a higher initial band than most blogs, and still a starting point to re-test. On this site the consistent start is about 500–800 tokens with ~100-token overlap (≈12–20%), then measure.

How do you decide chunk size in RAG?

Decide with constraints first, a starting band second, and a measured sweep last — never by copying a blog’s “optimal” number as law.

1. Respect the embedding model’s hard limit. Azure documents text-embedding-3-small at a maximum of 8191 tokens of input (Microsoft Learn; verify the current model card before you rely on it). Shorter open-source sentence transformers often hard-cap at 256 or 384 tokens and silently truncate anything longer (vinish.dev).
2. Fit the generator window. Retrieved tokens are roughly chunk_size × top_k (plus prompt overhead). If that product exceeds the LLM context you actually use, you are truncating answers at generation time (ByteTools 2026).
3. Pick a starting band, not an optimum. This site starts around 500–800 tokens with ~100-token overlap. Competing published peaks are corpus-specific: LlamaIndex’s October 2023 evaluation blog reported peak faithfulness and relevancy at 1024 tokens on Uber’s 2021 10-K across sizes {128, 256, 512, 1024, 2048}; Azure starts at 512 / 25%; several 2026 guides push ~512 as a pragmatic default. None of those numbers is “the” RAG optimum.
4. Sweep, holding everything else fixed. Index the same corpus at sizes {256, 512, 1024} × overlaps {0, 10%, 20%}, keep the embedding model, retriever, and top-k constant, and pick the cell by recall and token-level precision on your labelled questions. The experiment design and metrics live on how to evaluate a chunking strategy; many sweeps use fixed-size or recursive splitters as the baseline cell.

Query type leans the band but does not replace the sweep: factoid lookups often want smaller windows; analytical and narrative questions often want larger ones (Stackviv 2026; Digital Applied 2026). If traffic is mixed, segment by query class rather than forcing one size on everything.

Does the embedding model limit chunk size?

Yes — first as a hard ceiling, then as a soft optimum below that ceiling. Every embedding model has a maximum input length; text above it is truncated or rejected, so the tail of an oversized chunk never enters the vector (Microsoft Learn; Pinecone; Simplr on OpenAI embedding limits). Even below the max, longer chunks still average more topics into one representation, which is the dilution trade-off above. Match the chunker’s tokenizer to the embedding model’s tokenizer when you can, and leave headroom if you prepend document titles, section paths, or other metadata to the chunk text before embedding.

What failure does the wrong chunk size cause?

Wrong chunk size shows up as retrieval that feels broken when the documents are fine. Chunks that are too small return incomplete answers and unresolved references; chunks that are too large retrieve the right document with the wrong passage diluted in noise; seams without enough overlap (when your answers truly straddle boundaries) split one gold span across two chunk ids. Discriminating WHOLE / SPLIT / ABSENT and measuring answer-in-one-chunk rate belong on when the answer is split across two chunks; a gold span that is whole inside some indexed chunk but never retrieved is a wrong-chunk problem instead. Size and overlap change the odds — they do not remove the failure modes.