How to Evaluate a Chunking Strategy

Evaluating a chunking strategy means running a controlled experiment: fix the embedding model, the retriever and the top-k, change only the chunker, and score how well each version retrieves the answer on a labelled dataset. The chunker that fetches the answer with the least surrounding noise wins. Trying three questions by hand and deciding it “looks good” is not evaluation — it is how you ship a wrong-chunk bug you cannot explain later. This page is the experiment, end to end. It does not catalogue chunking strategies — that list lives at chunking; here you learn to measure them.

How do you build a labelled evaluation dataset?

You need questions each paired with the exact source span that answers it — the gold set. It is the foundation of the whole experiment: if the labels are sloppy, every number downstream is meaningless. There are two routes. Hand-label the answer span for each question in your own corpus — accurate, slow, and the standard for a benchmark you will trust. Or generate question/span pairs with an LLM and spot-check them — fast, but unverified pairs quietly corrupt the score. Use your own documents where you can; a public benchmark tells you which chunker wins on someone else’s corpus, not yours. Building eval datasets generalises well beyond chunking — see evaluation.

Which metrics separate a good chunker from a bad one?

Two, measured against the gold spans:

• Recall — did retrieval fetch the answer at all? Low recall means the right chunk never made it into the context; the model was doomed before it wrote a word.
• Precision — how little junk came with it? Low precision means the answer arrived buried in irrelevant text that the model has to see past.

Do not reduce this to “accuracy.” Accuracy hides where the failure is — recall and precision tell you whether to fix retrieval or fix the chunk. And there is a trap the Chroma research team named: measured at the chunk level, a chunker that returns the answer inside one enormous noisy chunk scores a perfect hit. So also measure at the token level — intersection-over-union of retrieved tokens against gold tokens — which exposes exactly how much noise rode along. The next section computes both on one example so the difference is concrete.

How do you compute precision and recall for a chunker?

Take one query Q. Suppose the gold set marks two chunks as truly relevant, {C1, C4}, and the retriever (top-3) returns {C1, C2, C3}. Count the outcomes: true positives = C1 (1); false positives = C2, C3 (2); false negatives = C4, the relevant chunk you missed (1). Then:

precision = TP / (TP + FP) = 1 / (1 + 2) = 0.33
recall    = TP / (TP + FN) = 1 / (1 + 1) = 0.50

Now the twist. Chunk C1 is 500 tokens, but the span that actually answers Q is only 40 tokens inside it. Chunk-level scoring already called C1 a perfect hit — yet you fed the model 460 tokens of noise to get 40 tokens of signal. The token-level metric makes that visible:

This is why a chunker can “pass” a chunk-level benchmark and still make retrieval feel noisy in production: oversized chunks inflate the chunk-level score while dragging token-level precision to the floor. Chronic low precision here is the same defect the wrong-chunk failure describes from the symptom side.

What chunk size and overlap should you test?

There is no universal best size — it depends on your content and your queries, which is the entire reason you run the experiment instead of copying a number off a blog. Rather than chase one “right” value, sweep a small grid and let the score decide: 256 / 512 / 1024 tokens × 0 / 10 / 20% overlap is a sensible first pass. Watch two traps at the ends: chunks too large bury the answer past a context cliff where the model stops attending to the middle; chunks too small sever a fact from the context that makes it meaningful. And re-test overlap rather than inheriting the default — extra overlap raises index size and cost, and often buys no recall. How size interacts with each splitter is covered under chunking; the defaults a first build should start from are at build a pipeline.

What mistakes invalidate a chunking benchmark?

The number is only as trustworthy as the experiment that produced it. Four things quietly invalidate it:

• Changing more than the chunker. Swap the embedding model or k at the same time and you no longer know what moved the score. Change one variable.
• A leaky gold set. Defining the “relevant” span after seeing which chunks were retrieved bakes the answer into the test. Label before you run.
• Micro-averaging blindly. Pooling every query lets one long document dominate the mean; report the macro average (per-query, then averaged) alongside it so a single doc cannot carry the result.
• Too few queries. Ten questions cannot separate two close chunkers. If the gap is within the noise, it is not a result.

What tools automate chunking evaluation?

You do not have to hand-roll the harness. Three real, neutral options — and this site sells none of them:

• Chroma’s chunking_evaluation — the package behind the token-level metrics used above; reach for it when you want that IoU rigour.
• messkan/rag-chunk — a CLI that benchmarks several chunkers on your corpus and prints a head-to-head; reach for it when you want a fast comparison, not a library to wire in.
• RAGAS — computes context precision and recall inside a broader RAG evaluation; reach for it when chunking is one part of an end-to-end eval.

The full neutral comparison of evaluation frameworks is at evaluation tools.