When the Answer Is Split Across Two Chunks

A chunk-boundary failure is when the gold answer string in the source straddles a chunk seam, so no single chunk contains it whole. Retrieval then returns a fragment, a near-miss neighbour, or nothing useful, and the model still answers fluently from half the evidence. Sphere Inc.’s RAG chunking guide (captured 18 September 2026) states the geometry plainly: if the right answer was sliced across two chunks or buried in a giant one, the system misses it no matter how strong the embeddings or the generator. Promptessor’s chunking guide (captured the same session) adds that a critical relationship split across a poor boundary leaves the retriever with only half the meaning. That looks like a retrieval bug. It is a chunking bug. It is not a returned wrong passage (why your RAG returns the wrong chunk) and not every other candidate-set miss (when RAG misses the document you know is there). Locate the answer in the source and classify the span before swapping embeddings. The why RAG systems fail taxonomy places this leaf under index and chunk.

How do you tell a chunk-boundary failure from a wrong chunk?

A wrong chunk means retrieval returned a passage that does not contain the answer. A chunk-boundary failure means the answer in the source crosses a seam, so even a perfect retriever can only hand the model a fragment, or neither fragment ranks high enough to enter top-k. Firecrawl’s 2026 chunking strategies guide frames the same ceiling: retrieval can only return a chunk, so a fact that never existed as a unit cannot surface cleanly. Run the span check first. Do not add a reranker until it says the gold answer sits wholly inside some indexed chunk.

Read the table as a triage card for one labelled failing query where you already know which source paragraph answers the question. Aggregate recall and context precision still matter for fleet health; they do not name whether this query failed because the answer was cut, because a wrong neighbour won, or because the document never entered the candidate set for another reason. Towards AI’s thin “stop blaming your LLM” framing (Rohan Mistry, captured 18 September 2026) points at chunking as a root cause class without shipping the three-way check that follows.

Once the span check returns SPLIT, the next question is what the failure looks like in the retrieved context the model actually saw, because that is the evidence operators notice before they open the source file.

What does a split-across-two-chunks failure look like?

A split-across-two-chunks failure looks like a retrieved context that starts the fact and never finishes it: the number, the exception, or the condition that makes the sentence an answer lives in the next chunk. Sphere’s opening example is an answer sliced across two chunks. Promptessor’s too-small case loses qualifiers and local context. Firecrawl’s overlap section describes a key sentence severed at a boundary so neither side alone is a complete thought. The model still writes a confident completion, because it only sees the fragment retrieval handed it.

Operators often notice the fluent wrong answer before they notice the seam. Databricks’ best-practice notes on incomplete sentence boundaries (Debu Sinha, captured 18 September 2026) treat chunk coherence as a quality signal for exactly this reason: a chunk that starts mid-clause or ends without terminal punctuation is a visible tell that the splitter cut on length. Promptessor’s anatomy of a good chunk requires that necessary local context stay with the fact; when that context is the exception line one paragraph later, the retrieved unit fails the well-scoped test even if cosine scores look healthy.

Symptoms alone still leave room for debate. The discriminating proof is the offset comparison that maps the gold span onto chunk ids.

How do you prove the answer straddles a chunk boundary?

You prove a straddling answer with offsets, not with vibes. Mark the gold answer span in the source, load each chunk’s start and end for that document, and classify. The named metric is answer-in-one-chunk rate: the fraction of labelled queries whose gold span sits wholly inside a single chunk. Sphere’s metadata section (captured 18 September 2026) stores each chunk’s title, page, and exact character offsets specifically so an answer can point back to the exact source region; those same offsets are what make the span classifier honest against the live index.

Store the same coordinate system you split on. Character offsets against Markdown that was later rendered to HTML will false-positive SPLIT. Token offsets against a character chunker will do the same. When chunks live in Weaviate, Pinecone, Qdrant, or another vector store, persist start, end, and source_id as properties at ingest so the classifier reads production boundaries rather than a rebuilt offline cut.

Labelling fifty to one hundred failing queries is enough to see whether SPLIT dominates a corpus type. Firecrawl’s decision framework still starts teams on recursive defaults; the span class is what tells you whether that default left answers whole. Promptessor’s evaluation advice (measure retrieval and answer quality after chunking changes) is the surrounding loop; WHOLE/SPLIT/ABSENT is the chunking-specific meter inside it.

Measurement is mandatory because the failure is silent. Chroma’s 2024 chunking evaluation (trychroma.com/research/evaluating-chunking) shows why size alone is not a free fix: on their text-embedding-3-large table, recursive splitting at 200 tokens with 0 overlap reached about 7.0% mean token-level precision, while recursive at 800 tokens with 400-token overlap fell to about 1.5%. Larger windows drown the relevant tokens even when recall stays high. Use those figures as evidence that you must measure on your spans, not as a universal target. Databricks’ evaluation section (Debu Sinha, captured 18 September 2026) defines a related concept-integrity idea: how often key phrases are split across chunks. The WHOLE/SPLIT/ABSENT class is the binary that isolates this failure before any strategy bake-off.

Once SPLIT is proven, the usual cause is a length-based splitter that ignored the document’s own seams.

Why does fixed-size chunking cut answers in half?

Fixed-size chunking cuts answers in half because it splits every N tokens or characters regardless of paragraph, sentence, or section boundaries. A 512-token window can cut mid-sentence, mid-table, or between a rule and its exception. The embedding then represents a fragment, and retrieval cannot surface a complete answer that never existed as a unit. Firecrawl’s size-based section states the trade-off: character or token counting is simple and can still split tables, code blocks, or lists in awkward places. Sphere calls fixed-size the default that quietly wrecks accuracy on structured documents because it cuts blind through a table, between a heading and the paragraph it governs, or mid-clause. Promptessor treats fixed-size as a useful baseline that becomes weak when the source has meaningful structure the splitter ignores. Databricks’ fixed-size example uses a 1000-character window with 200-character overlap as an illustration, not as a law.

Token-based fixed splits are not safer just because they match embedding limits. Firecrawl notes that tokenizers and characters diverge (one word can be multiple tokens), so a “512” target means different prose coverage than a 512-character cut. Either way, the splitter is still length-first. The only way length-first chunking stops producing SPLIT on a given answer is if that answer happens to fit inside one window after the cut, which is luck, not a design guarantee.

Teams often assume a recursive or structure-aware splitter ends the failure. It reduces mid-word cuts; it does not guarantee WHOLE when the answer outruns the remaining room in a chunk.

Does recursive or structure-aware splitting stop mid-answer cuts?

Recursive and structure-aware splitting reduce mid-answer cuts; they do not stop every SPLIT. Firecrawl treats recursive character splitting as the default start for most text: separators try paragraph breaks, then lines, then sentences, then spaces, so a 600-character paragraph under a 512 target splits at a sentence rather than mid-word. Their worked default is a 400 to 512 token-scale window with 10 to 20 percent overlap for general use. Databricks’ recursive section and Promptessor’s recursive and structure-aware sections make the same claim: respect natural boundaries before falling back to length. Sphere’s production default for enterprise content is structure-aware splitting along headings, sections, paragraphs, list items, table rows, and contract clauses so each chunk is a coherent unit of meaning.

The honesty line is mechanical. Recursive separators only choose where to cut inside the length budget. If the gold answer is longer than the unused capacity left in the current chunk, the splitter still emits a boundary inside the answer. Structure-aware splitting helps when the author already put the answer inside one section or clause; it cannot invent a single unit for a fact that genuinely spans two sections. Dev.to’s Vishal Mysore deep dive (captured 18 September 2026) lists sliding-window and adaptive strategies that still carry size and overlap knobs, which is another way of saying boundaries remain a design choice. Mechanism depth for separator hierarchies lives under recursive chunking and structure-aware chunking; the failure question here is only whether SPLIT still appears after those seams are in place.

Sentence-based splitters, which Firecrawl and Databricks both discuss as a related family, keep sentence integrity and still fail the same length test: an answer that is three sentences long under a one-sentence budget becomes three retrieval atoms, and the query that needs all three still classifies as SPLIT unless overlap or a parent reunites them. Page-level chunking avoids mid-page cuts and creates a different problem when a page mixes unrelated sections. Those variants belong in the strategy catalogue; the diagnostic here stays the span class.

When structure-aware seams are already in use and short straddling spans remain, the next lever people reach for is overlap. Overlap only works when the answer is shorter than the repeated window.

Does chunk overlap fix a boundary split?

Chunk overlap fixes a boundary split only when the gold answer is shorter than the overlap window. Overlap repeats seam tokens so a short straddling span can land wholly inside at least one chunk. If the answer is longer than the overlap, both chunks still hold fragments: you have paid for duplicates without converting SPLIT to WHOLE. Firecrawl states that industry practice often starts at 10 to 20 percent overlap, for example 50 to 100 tokens on a 500-token chunk, and that more overlap preserves more context while raising storage and processing cost. Databricks’ best-practice bands (Debu Sinha) put general text near 200 to 500 tokens with 10 to 20 percent overlap, and code-like content near 100 to 200 tokens with 15 to 25 percent overlap. On this site the consistent starting point is about 500–800 tokens with roughly 100-token overlap, then measure. Sweep grids such as 256 / 512 / 1024 × 0 / 10 / 20% belong on the page that measures chunk size, not as a universal law here.

The decision rule is the discriminating gain the strategy catalogues leave unstated as a binary: compare len(gold_span) to chunk_overlap in the same units. Sphere’s overlap section adds the production warning that heavy overlap inflates the index, returns near-duplicate chunks that crowd out different results, and wastes generation tokens, usually a sign boundaries are wrong rather than that more overlap is needed. Extend AI’s semantic chunking practices (March 2026) still recommend overlap windows as a best practice after threshold tuning, which is consistent with treating overlap as a seam repair, not as a substitute for parent retrieval.

Overlap is also contested as a free win. Firecrawl cites Bennani and Moslonka’s January 2026 systematic analysis (arXiv 2601.14123), which used SPLADE retrieval and Mistral-8B on Natural Questions and found that overlap provided no measurable benefit and only increased indexing cost on that setup. That finding does not ban overlap; it bans copying a percentage without measuring answer-in-one-chunk rate and retrieval metrics on your corpus. Extra overlap grows the index and produces near-duplicate hits; Chroma’s IoU-style penalties in chunking research explicitly punish redundant overlap tokens. When the gold span is longer than any affordable overlap window, the architecture has to change: retrieve on a small unit and generate from a larger parent.

When do you need hierarchical or parent-document chunking?

You need hierarchical or parent-document chunking when SPLIT survives structure-aware boundaries and an overlap window that already covers short straddling spans. The architecture move is the same across frameworks: embed and retrieve on small units for sharp ranking, then hand the model the larger parent, or a window of neighbouring sentences, at generation time. Promptessor’s parent-child section states the pattern as matching on small chunks while returning a larger parent section for generation. Sphere’s hierarchical section keeps multiple granularities and the links between them: small chunks for precise retrieval plus their parent section for context, which fits long manuals, regulations, and contracts where one size cannot serve both precision and context. Dev.to’s parent-child strategy is the same retrieve-small, expand-parent idea under a different label.

Index cost is real. Parent retrieval stores more vectors or a second lookup path, and a re-chunk with new parents forces a re-embed of the corpus. Promptessor’s cost and latency section treats chunk count as a driver of embedding spend and retrieval latency; hierarchical designs raise that count on purpose because precision at retrieve time and completeness at generate time are different jobs. That cost is still cheaper than permanent SPLIT on the questions users actually ask, once the span-length check has ruled overlap out.

Parent ids belong in chunk metadata the same way offsets do. In Weaviate, Qdrant, Elasticsearch, or LlamaIndex-style node graphs, each child keeps a parent_id (or equivalent) so retrieval can expand without a second blind search. When teams ask whether semantic breakpoints remove the need for parents, the honest answer is that semantic chunking attacks the same failure by moving boundaries, not by expanding context after retrieval.

Does semantic chunking prevent boundary loss?

Semantic chunking reduces boundary loss; it does not remove it. It splits where neighbouring sentence embeddings diverge, so coherent ideas tend to stay together, which is exactly the failure named here. A wrong similarity threshold still cuts mid-idea, and a fact that genuinely spans a topic change can still land in two chunks. Firecrawl’s semantic section cites the NAACL 2025 Findings paper (Anthology 2025.findings-naacl.114; also arXiv 2410.13070, Qu et al.) concluding that the computational costs are not justified by consistent gains, with fixed roughly 200-word chunks matching or beating semantic chunking across retrieval and answer generation on their tasks. That is a published finding on their setup, not a ban.

Extend AI’s March 2026 practices still recommend domain-appropriate embeddings, tuned similarity thresholds, overlap windows, metadata tags, and monitored metrics for semantic chunking in production document processing. Promptessor and Databricks both list semantic or meaning-based splits as a strategy that keeps related sentences together. Use semantic chunking when your corpus is heterogeneous, and prove it on how to evaluate a chunking strategy rather than assuming the label “semantic” ends SPLITs. Late chunking and contextual retrieval, which enrich embeddings with document-level context, are owned by late and contextual chunking; they are not substitutes for proving WHOLE on your gold spans.

Firecrawl’s cost spectrum note still applies after a semantic trial: embedding every sentence to find breakpoints multiplies ingest spend relative to recursive defaults. If answer-in-one-chunk rate barely moves while ingest latency rises, the experiment has answered itself. Even with better sentence seams, some corpora make boundary loss worse by construction: contracts, manuals, and tables whose unit of meaning is not a paragraph.

Why do structured documents make boundary loss worse?

Structured documents make boundary loss worse because the unit a human asks about is often a clause, a procedure step, a table row with its header, or a code block, not an arbitrary token window. Sphere’s document-type guidance is explicit: contracts and legal text should chunk by clause, because splitting one obligation across chunks returns half an obligation; technical manuals and SOPs need hierarchical small steps plus parent procedures so “step 4” does not arrive without the procedure; support tickets should stay per thread because over-splitting destroys conversation context. Promptessor’s content-type sections for tables, code, PDFs, and transcripts make the same point: type-aware splits keep the natural unit intact. Databricks’ hybrid processing example keeps tables intact rather than treating pipe-delimited rows as prose.

Code is a sharp special case of the same geometry. Databricks cites a MongoDB suggestion that language-specific recursive splitters around roughly 100 tokens with about 15 tokens of overlap often balance precision and recall on Python documentation. Dev.to’s code-specific strategy similarly refuses to treat source like prose. Those starting bands are not laws; they are reminders that a function signature and its body are one answer unit the way a contract clause is. When flattening destroys the unit column or the header that makes a number answerable, the dedicated failure is why RAG gets numbers and tables wrong, not a generic overlap tweak.

The lean claim here is only diagnostic: if gold spans on structured sources classify as SPLIT far more often than prose spans on the same corpus, change the splitter’s unit of meaning before you raise overlap again. Once structure risk is clear, the operational meter that decides whether chunking can carry answers at all is the answer-in-one-chunk rate.

How do you measure answer-in-one-chunk rate?

You measure answer-in-one-chunk rate by classifying each labelled gold span against chunk offsets and counting WHOLE over N. Each labelled item is a query plus the gold answer’s start and end in the source. Anything still SPLIT after you change the chunker is the regression signal to watch. Databricks’ concept-integrity metric asks a related question (how often key phrases split); the classifier below answers the binary that isolates this failure. Build the labelled set first. Without gold offsets, the function cannot name SPLIT.

A practical labelling pass starts from production failures, not from random paragraphs. Take queries where the answer was wrong or incomplete, open the source, highlight the minimal span a correct answer needs, and record its offsets against the text that was actually chunked. Promptessor’s checklist item on evaluating only retrieval similarity is the trap this meter avoids: high similarity on a fragment still scores as a retrieval hit while the span class remains SPLIT. Keep ABSENT rows in the same file so ingest regressions do not get misfiled as boundary bugs.

def classify_span(span_start, span_end, chunks):
    """Classify a gold answer span against chunk offsets.

    All offsets are half-open [start, end) in the same units (chars or tokens).
    chunks: iterable of dicts with keys id, start, end
    Returns: "WHOLE" | "SPLIT" | "ABSENT"
    """
    covering = [c for c in chunks if c["start"] <= span_start and span_end <= c["end"]]
    if covering:
        return "WHOLE"
    touching = [c for c in chunks if c["start"] < span_end and span_start < c["end"]]
    if not touching:
        return "ABSENT"
    last = span_end - 1  # last included unit of the gold span
    start_ids = {c["id"] for c in touching if c["start"] <= span_start < c["end"]}
    end_ids = {c["id"] for c in touching if c["start"] <= last < c["end"]}
    if start_ids and end_ids and start_ids == end_ids:
        return "WHOLE"
    return "SPLIT"


def answer_in_one_chunk_rate(labelled, chunks_by_doc):
    """labelled: list of {doc_id, span_start, span_end}
    chunks_by_doc: doc_id -> list of chunk dicts
    """
    labels = [
        classify_span(row["span_start"], row["span_end"], chunks_by_doc[row["doc_id"]])
        for row in labelled
    ]
    whole = sum(1 for x in labels if x == "WHOLE")
    rate = whole / len(labels) if labels else 0.0
    print(f"answer-in-one-chunk rate: {rate:.0%}  ({whole}/{len(labels)} WHOLE)")
    print({k: labels.count(k) for k in ("WHOLE", "SPLIT", "ABSENT")})
    return rate


# Expected shape of output on a 20-query set with 14 WHOLE / 5 SPLIT / 1 ABSENT:
# answer-in-one-chunk rate: 70%  (14/20 WHOLE)
# {'WHOLE': 14, 'SPLIT': 5, 'ABSENT': 1}

Once you can name SPLIT, tune with chunking evaluation and keep the rate from regressing in regression testing. The metric tells you whether chunking can carry the answer; it does not replace retrieval metrics once the span is WHOLE. Wire the classifier against the same Weaviate collection (or other store) you serve in production so the printed class matches live boundaries rather than a toy re-split.

A chunk-boundary failure is not a mysterious retrieval miss. It is a gold answer that never existed as a single indexed unit: classify WHOLE, SPLIT, or ABSENT from offsets, treat overlap as a fix only when the span fits inside the overlap window, and escalate to hierarchical or parent-document retrieval when it does not. Measure answer-in-one-chunk rate on a labelled set, change one boundary lever at a time, and keep the failing spans in regression so the next re-chunk cannot silently recreate the same seam. The consequence is operational: until WHOLE dominates on the questions users ask, retrieval tuning and prompt changes are optimizing around a cut that already destroyed the evidence.