Chunk Size and Overlap: How to Choose Them

Chunk size is the maximum length of each passage you embed for retrieval, usually counted in tokens, and there is no universal best value across corpora. Small chunks raise precision for short factual answers and raise the odds that a gold span is split across a boundary; large chunks keep more context for generation and dilute the embedding toward a blur of topics. SurrealDB’s recommended-chunk-size guide (captured 18 September 2026) states the common general-purpose band as 512 to 1,024 tokens with 10% to 20% overlap, then tells readers to measure rather than copy the band forever. Weaviate’s chunking strategies blog (as of September 2026) publishes the same 10% to 20% overlap starting point for fixed-size work. The parent map for strategy choice is how to split documents that retrieve well; this leaf owns the length and overlap knobs every splitter still exposes.

What is chunk size in RAG?

Chunk size in RAG is the length budget for each indexed passage: the maximum tokens (or characters, if the splitter counts characters) allowed in one unit before the next unit begins. SurrealDB (18 September 2026) measures size in tokens and notes that a token is roughly four characters in English, so 512 tokens is about 2,000 characters or roughly three to four paragraphs. Embedding models also have their own maximum sequence lengths; text past that limit is truncated at embed time, which is a hard ceiling separate from the soft budget you choose for retrieval quality.

Overlap is a related but distinct knob: some tokens from the end of one chunk are repeated at the start of the next so a sentence that straddles a seam can still appear whole in at least one unit. Size decides how much meaning one vector represents; overlap decides how much seam insurance you buy with duplicated storage. Both are configuration, not a strategy identity: fixed-size, recursive and structure-aware splitters all still take a size (and often an overlap) argument.

Once size is defined as a budget, the next question is how that budget changes what retrieval returns.

How does chunk size affect retrieval quality?

Chunk size affects retrieval quality by trading precision against context: smaller units match short queries more specifically, while larger units keep surrounding sentences the generator needs and average more topics into one vector. Stack Overflow’s 27 December 2024 chunking article quotes Pinecone’s Roie Schwaber-Cohen on the mismatch risk: when embedded content is wildly larger than the query, similarity scores fall and the retriever hits the wrong unit. Weaviate’s sweet-spot framing (September 2026) is the same trade without a universal percentage: chunks must be specific enough to retrieve and complete enough to generate.

Published experiments disagree on a single winner because corpora disagree. LlamaIndex’s ideal-chunk-size blog evaluates 128, 256, 512, 1024 and 2048 on questions generated from the first 20 pages of an Uber 10-K and reports average faithfulness peaking at chunk size 1024, with average relevancy also peaking at 1024 on that run, while response time rises only slightly as size grows. That peak is evidence for that document and that evaluator stack, not a law for every PDF. Towards Data Science’s character-budget experiment (80, 220 and 500 characters with overlap) describes 80-character units as fragmented sentence pieces, 220 as apparently stable, and 500 as more coherent context, again on one small system rather than a public leaderboard.

ArXiv paper 2505.21700 (“Rethinking Chunk Size for Long-Document Retrieval,” HTML version captured 18 September 2026) reports a multi-dataset fixed-size study: smaller chunks (64 to 128 tokens) suit concise fact-based answers, while larger chunks (512 to 1024 tokens) help when answers need broader context. On TechQA it reports Snowflake recall@1 rising from 4.8% at 64 tokens to 71.5% at 1024 tokens. On SQuAD it reports recall@1 already high at 64 tokens (64.1%) with only marginal gains from larger sizes. Stella and Snowflake also disagree on COVID-QA peaks (Stella highest recall@1 52.1% at 64 tokens; Snowflake peaking at 1024 tokens with recall@1 54.2% in the paper’s report). Those figures are the paper’s measurements under its string-matching evaluation; they are not this site’s proprietary bake-off.

Size alone does not name the seam problem. Overlap is the knob that tries to keep straddling sentences inside at least one chunk.

What does chunk overlap do?

Chunk overlap repeats a trailing window of tokens at the start of the next chunk so content near a boundary is not stranded in a unit the retriever never ranks. Weaviate (September 2026) describes overlap as the common fix when fixed-size splits cut mid-sentence: some tokens from the end of one chunk are repeated at the beginning of the next. SurrealDB frames the same idea as preserving continuity across boundaries when key sentences would otherwise split.

Overlap is not free. Every overlapped token is stored and embedded twice, which raises index size and can surface near-duplicate neighbours in top-k. Stack Overflow notes the storage and redundancy cost of sliding windows explicitly. Overlap also does not repair a unit that was never the right granularity: a multi-topic blob with 15% overlap is still a diluted blob. When the gold answer is longer than the overlap window and still straddles a seam, the next repair is usually hierarchical parent return rather than endless overlap growth, which is owned by hierarchical chunking after the symptom is confirmed.

Knowing what overlap does still leaves the numeric band open. The next question is how much overlap to start with, and when that band is only a rumour.

How much chunk overlap should you use?

A common starting overlap is 10% to 20% of chunk size for general document Q&A, treated as a prototype default to measure rather than a proven optimum. Weaviate’s fixed-size guidance (September 2026) states a typical overlap between 10% and 20% of chunk size. SurrealDB’s table (18 September 2026) lists 10% to 20% for general RAG, about 50 to 100 tokens as an example absolute window, 10% to 15% for long technical docs, and none for short FAQ-style units or per-function code chunks.

Overlap that is copied without measurement is a defect waiting to happen. If failing queries show the gold string split across two consecutive chunks, SurrealDB’s tuning step is to raise overlap toward 15% to 20% and re-score. If metrics do not move, the boundary problem is larger than the window and belongs with parent-document retrieval or structure-aware cuts. Competing posts sometimes crown a single overlap percent; this leaf keeps the band attributed to named guides and routes the controlled comparison to the evaluation leaf for this cluster.

Size and overlap are still only knobs. The decision procedure is how to pick starting values for a real corpus without waiting for a vendor to bless one number.

How do you decide chunk size in RAG?

You decide chunk size by matching document type and answer locality, picking a published starting band, then sweeping neighbouring sizes on a labelled set while holding the embedder and retriever fixed. SurrealDB’s use-case table (18 September 2026) is a practical start: 512 to 1,024 tokens with 10% to 20% overlap for general Q&A; 128 to 256 with no overlap for short FAQs; 1,024 to 2,048 with 10% to 15% overlap for long technical docs; per-function or class units for code. ArXiv 2505.21700’s dataset split agrees in direction: factoid Wikipedia-style answers tolerate small chunks; explanation-heavy technical answers need larger ones.

A minimal sweep grid that stays honest is three sizes around your baseline (for example 256, 512 and 1,024 tokens) crossed with three overlap settings (0%, 10% and 20%), scored with the same gold questions. LlamaIndex’s blog demonstrates the size axis with faithfulness and relevancy evaluators; the full harness that isolates the chunker from the embedder lives on the evaluation leaf. Promote a cell only when the lift clears the re-embed cost. Do not invent a fourth magic number because a single 10-K peak looked good in someone else’s notebook.

Store the winner beside the embedding model name and the date of the run. Volatile defaults in splitter libraries move; a size that won in March can lose after a model swap. Two constraints still sit above taste: the embedding model’s maximum sequence length, and the failure mode wrong size creates in production.

Does the embedding model limit chunk size?

Yes. The embedding model’s maximum sequence length is a hard ceiling: tokens past the limit are truncated and never enter the vector, so a “chunk size” larger than the model window is partly fiction. SurrealDB (18 September 2026) cites model windows typically from 512 to 8,192 tokens depending on the model, and names OpenAI text-embedding-3-small as supporting up to 8,191 tokens while noting shorter inputs often retrieve better. Weaviate’s production guidance treats chunk size as something you tune inside the model’s usable window, not past it.

Soft limits matter too. Very long chunks that still fit the window can blur many ideas into one embedding, which is the dilution failure Schwaber-Cohen describes. Multivector or late-chunking setups change how context is preserved without simply raising the character budget; those upgrades are owned by late chunking and contextual retrieval. When two embedding models disagree on the best size, as Stella versus Snowflake do in arXiv 2505.21700, the bake-off must name the embedder version beside the chunk size.

Limits explain truncation. Wrong size still produces a visible retrieval failure even when nothing was truncated.

What failure does the wrong chunk size cause?

Wrong chunk size causes wrong-chunk retrieval and chunk-boundary loss: either the returned unit is near the answer but incomplete or diluted, or the answer string never sits inside any single chunk. AWS and practitioner debug guides on the wrong-chunk leaf treat granularity as one of five causes with answer-in-one-chunk rate as the discriminating meter. Too-small splits cut dollar amounts and closing sentences out of top-k; too-large splits bury the fact inside unrelated paragraphs so cosine neighbours look healthy and still fail a human read.

The symptom page for a confidently wrong retrieved passage is why your RAG returns the wrong chunk. The symptom page for an answer straddling a seam is when the answer is split across two chunks. Fixing prompts will not rejoin a split span. Fixing size and overlap, or escalating to hierarchical return, will. Measure before you re-chunk the fleet: a labelled set of 25 to 50 question and expected-answer pairs is enough to start, as SurrealDB’s test protocol states, and the scored comparison across strategies is chunking evaluation.

Chunk size and overlap are the length knobs on every splitter, not a substitute for choosing the right strategy family. SurrealDB and Weaviate publish overlapping starting bands (hundreds to about a thousand tokens, often with 10% to 20% overlap); LlamaIndex and arXiv 2505.21700 show that peaks move with corpus and embedder. Start in a documented band, sweep neighbours on gold spans, and treat truncation limits as hard stops. The wrong size shows up as retrieval, and only a measured change to the length budget repairs it on the next full index build.