When the Answer Is Split Across Two Chunks

A chunk-boundary failure is when the answer string in your source straddles a chunk seam — so no single chunk contains it whole. Retrieval then returns a fragment, a near-miss neighbour, or nothing useful, and the model answers fluently from half the evidence. It looks like a retrieval bug. It is a chunking bug. It is not the case where a chunk came back and was simply the wrong passage (see wrong chunk), and it is not every other reason a known document never becomes a candidate (see missing document). Before you swap embeddings or add a reranker, locate the answer in the source and classify the span.

The failures hub pins this under index and chunk. The job here is to prove the split, decide whether overlap is enough, and know when you need hierarchical / parent-document chunking instead.

How do you tell a chunk-boundary failure from a wrong chunk?

A wrong chunk means retrieval returned a passage that does not contain the answer. A chunk-boundary failure means the answer in the source crosses a seam, so even a perfect retriever can only hand the model a fragment — or neither fragment ranks. Run the span check first; do not add a reranker until it says the gold answer sits wholly inside some indexed chunk.

What does a split-across-two-chunks failure look like?

The retrieved context starts the fact and never finishes it — the number, the exception, or the condition that makes the sentence an answer lives in the next chunk. Practitioners describe half-sentences, broken context, and “the rule without the exception.” The model still writes a confident completion.

How do you prove the answer straddles a chunk boundary?

Prove it with offsets, not vibes. Mark the gold answer span in the source, load each chunk’s start and end for that document, and classify. The named metric is answer-in-one-chunk rate: the fraction of labelled queries whose gold span sits wholly inside a single chunk.

Measurement is mandatory because the failure is silent. Chroma’s 2024 chunking evaluation (trychroma.com/research/evaluating-chunking) shows why size alone is not a free fix: on their text-embedding-3-large table, recursive splitting at 200 tokens with 0 overlap reached about 7.0% mean token-level precision, while recursive at 800 tokens with 400-token overlap fell to about 1.5% — larger windows drown the relevant tokens even when recall stays high. Use those figures as evidence that you must measure on your spans, not as a universal target.

Why does fixed-size chunking cut answers in half?

Fixed-size chunking splits every N tokens or characters regardless of paragraph, sentence, or section boundaries. A 512-token window can cut mid-sentence, mid-table, or between a rule and its exception. The embedding then represents a fragment, and retrieval cannot surface a complete answer that never existed as a unit.

Does chunk overlap fix a boundary split?

Chunk overlap fixes a boundary split only when the gold answer is shorter than the overlap window. Overlap repeats seam tokens so a short straddling span can land wholly inside at least one chunk. If the answer is longer than the overlap, both chunks still hold fragments — you have paid for duplicates without converting SPLIT to WHOLE.

Published starting bands converge on roughly 10–20% of chunk size (Ertas 2026; CustomGPT guidance; Bswen’s worked example uses 200 overlap on a 1000-unit chunk ≈ 20%). On this site the consistent starting point is about 500–800 tokens with ~100-token overlap — then measure. Sweep grids such as 256 / 512 / 1024 × 0 / 10 / 20% belong on chunk size and overlap, not as a universal law here. Extra overlap grows the index and produces near-duplicate hits; Chroma’s IoU metric explicitly penalises redundant overlap tokens.

When do you need hierarchical or parent-document chunking?

You need hierarchical or parent-document chunking when SPLIT survives structure-aware boundaries and an overlap window that already covers short straddling spans. The architecture move is the same across frameworks: embed and retrieve on small units for sharp ranking, then hand the model the larger parent — or a window of neighbouring sentences — at generation time.

Does semantic chunking prevent boundary loss?

Semantic chunking reduces boundary loss; it does not remove it. It splits where neighbouring sentence embeddings diverge, so coherent ideas tend to stay together — which is exactly the failure this page names. A wrong similarity threshold still cuts mid-idea, and a fact that genuinely spans a topic change can still land in two chunks.

Qu et al., Findings of NAACL 2025 (arXiv 2410.13070), tested semantic chunking against fixed-size chunking across document retrieval, evidence retrieval, and answer generation, and reported that the extra compute was not justified by consistent performance gains. That is a published finding on their setup, not a ban: use semantic chunking when your corpus is heterogeneous, and prove it on a controlled chunking experiment rather than assuming the label “semantic” ends SPLITs.

How do you measure answer-in-one-chunk rate?

Run the classifier over a labelled set: each item is a query plus the gold answer’s start and end offsets in the source. The rate is the share classified WHOLE. Anything still SPLIT after you change the chunker is the regression signal to watch.

def classify_span(span_start, span_end, chunks):
    """Classify a gold answer span against chunk offsets.

    All offsets are half-open [start, end) in the same units (chars or tokens).
    chunks: iterable of dicts with keys id, start, end
    Returns: "WHOLE" | "SPLIT" | "ABSENT"
    """
    covering = [c for c in chunks if c["start"] <= span_start and span_end <= c["end"]]
    if covering:
        return "WHOLE"
    touching = [c for c in chunks if c["start"] < span_end and span_start < c["end"]]
    if not touching:
        return "ABSENT"
    last = span_end - 1  # last included unit of the gold span
    start_ids = {c["id"] for c in touching if c["start"] <= span_start < c["end"]}
    end_ids = {c["id"] for c in touching if c["start"] <= last < c["end"]}
    if start_ids and end_ids and start_ids == end_ids:
        return "WHOLE"
    return "SPLIT"

def answer_in_one_chunk_rate(labelled, chunks_by_doc):
    """labelled: list of {doc_id, span_start, span_end}
    chunks_by_doc: doc_id -> list of chunk dicts
    """
    labels = [
        classify_span(row["span_start"], row["span_end"], chunks_by_doc[row["doc_id"]])
        for row in labelled
    ]
    whole = sum(1 for x in labels if x == "WHOLE")
    rate = whole / len(labels) if labels else 0.0
    print(f"answer-in-one-chunk rate: {rate:.0%}  ({whole}/{len(labels)} WHOLE)")
    print({k: labels.count(k) for k in ("WHOLE", "SPLIT", "ABSENT")})
    return rate

# Expected shape of output on a 20-query set with 14 WHOLE / 5 SPLIT / 1 ABSENT:
# answer-in-one-chunk rate: 70%  (14/20 WHOLE)
# {'WHOLE': 14, 'SPLIT': 5, 'ABSENT': 1}

Once you can name SPLIT, tune with chunking evaluation and keep the rate from regressing in regression tests. The metric tells you whether chunking can carry the answer; it does not replace retrieval metrics once the span is WHOLE.