Lost in the Middle: When the Right Chunk Is Ignored

The right chunk is in the prompt — and the model still answers as if it were not. That is lost in the middle: retrieval succeeded, but position bias caused the generator to under-use the middle of a long context. It is not a wrong chunk (the gold text never came back), not a missing document (nothing relevant was ever retrieved), and not conflicting sources (two passages disagree). Confirm the gold passage is inside the assembled prompt before you touch the retriever.

Lost-in-the-middle is a generation-stage failure over a context that retrieval already filled. The fix is almost always how you order and truncate that context — not a new embedding model.

How do you know it is lost in the middle?

You know it is lost in the middle when the gold chunk is present in the prompt and moving that same chunk from the middle to the start (or end) measurably improves the answer. Other retrieval failures look similar from the outside — so run one discriminating check before you reorder anything.

Debugging guides often jump straight to “reorder your chunks.” That only helps when the position delta is real. If the gold passage is not in the prompt at all, reordering cannot recover it — leave this page and open the matching failure. For the full stage walk, see detection.

What did Liu et al. measure about context position?

Liu et al. (2023; TACL 2024) measured that multi-document question-answering accuracy follows a U-shaped curve: models use a relevant document best when it sits at the beginning or the end of the input context, and significantly worse when it sits in the middle — even for models marketed as long-context.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang published the result as Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172; TACL 2024). They controlled context length with 10, 20 and 30 documents and moved the single answer-bearing document through positions from start to end. On multi-document QA, GPT-3.5-Turbo’s accuracy can drop by more than 20 percentage points when the relevant document moves to the middle; in the worst 20- and 30-document settings, middle-position performance fell below the model’s closed-book baseline of 56.1% — answering with no documents at all. Their closed-book versus oracle (single gold document) baselines were: GPT-3.5-Turbo 56.1% / 88.3%; GPT-3.5-Turbo (16K) 56.0% / 88.6%; Claude-1.3 48.3% / 76.1%; LongChat-13B (16K) 35.0% / 83.4%; MPT-30B-Instruct 31.5% / 81.9% (Liu et al., Table 1).

In an open-domain NaturalQuestions-Open case study, reader performance saturated long before retriever recall: using more than 20 retrieved documents improved GPT-3.5-Turbo by only about 1.5 percentage points and Claude-1.3 by about 1 percentage point, while context length (and cost) kept rising (Liu et al., §5). Those are the published figures — do not treat blog reconstructions that invent a middle-position percentage (for example “~65%”) as Liu’s numbers.

Why do language models ignore the middle of the prompt?

Language models ignore the middle of the prompt because decoder-only transformers exhibit primacy bias (stronger use of early tokens) and recency bias (stronger use of late tokens), so middle positions receive less effective use even when every token is technically in the window.

Liu et al. document the behavioural U-curve; follow-up work attributes the pattern to how causal attention and positional encodings (commonly RoPE) weight sequence positions. This page stays on the failure you can measure in a RAG prompt. Architectural mitigations — multi-scale positional encodings, attention calibration, specialised long-context training — belong on where to put the best chunk, not here. Embedding choice did not cause this failure: embeddings decide what gets retrieved; position decides what gets attended to once it is in the prompt.

Does a longer context window fix lost in the middle?

A longer context window does not fix lost in the middle. Liu et al. found that when the same 10- or 20-document inputs fit both a base model and its extended-context counterpart (for example GPT-3.5-Turbo versus GPT-3.5-Turbo 16K), the position curves were nearly superimposed — extending the window did not remove the U-shape. A bigger window mostly adds more middle.

That matches the open-domain finding above: past roughly 20 retrieved documents, extra recall barely moved answer accuracy for the models they tested. Databricks’ long-context RAG blog (12 August 2024) later reported a related production pattern on their suites: Llama-3.1-405B quality began decreasing after 32k tokens and GPT-4-0125-preview after 64k tokens — longer was not uniformly better. Treat those Databricks thresholds as their measured curves, not universal constants; retune on your corpus. The practical control is how many chunks you retrieve, not the advertised maximum context length.

How do you fix lost in the middle in a RAG prompt?

You fix lost in the middle by placing the highest-relevance chunks at the start and end of the context, putting weaker chunks in the middle, and cutting top-k until extra documents stop helping the answer. Fighting the U-curve is less reliable than working with it.

Four production moves, in the order most teams should try them:

1. U-shape the order after you score relevance. Score candidates with a cross-encoder reranker, then place the best chunk first, the second-best last, and the weakest in the middle. For five chunks ranked A≻B≻C≻D≻E, a typical edge-first layout is A, C, E, D, B — A and B on the edges, E in the centre. LangChain’s LongContextReorder implements the same idea; the algorithm matters more than the framework. Full ordering mechanics: context ordering.
2. Cut top-k. Liu et al. saw only about 1.5 percentage points (GPT-3.5-Turbo) and about 1 percentage point (Claude-1.3) of reader gain past 20 documents while recall was still rising. Prefer a short, edge-heavy window over a long middle. Tune k on choosing top-k.
3. Query-aware contextualization. Place the user question both before and after the documents so the model can condition on the query while reading. Liu et al. (§4.2) found this made key-value retrieval near-perfect — GPT-3.5-Turbo (16K) reached perfect accuracy at 300 key-value pairs — while without it the worst-case key-value accuracy was 45.6%. On multi-document QA the same trick barely moved the U-curve; state that honestly when you adopt it.
4. An explicit “use every passage, including the middle” instruction can help at the margin. It is not a substitute for reorder and cut-k. Prompt compression and multi-pass extraction are valid deeper mitigations — see context ordering and RAG prompts.

How do you run a position test on your own pipeline?

You run a position test by taking one failing query, the chunk you know contains the answer, and a handful of distractors, then asking the same model three times with the gold chunk at the first, middle and last positions. A middle miss with a first/last hit is the lost-in-the-middle signature.

def place(gold, distractors, position):
    """Return chunks with `gold` at first | middle | last among distractors."""
    others = list(distractors)
    if position == "first":
        return [gold] + others
    if position == "last":
        return others + [gold]
    mid = len(others) // 2
    return others[:mid] + [gold] + others[mid:]

def position_test(query, gold, distractors, ask, *, judge=None):
    """ask(prompt) -> answer string. Optional judge(answer) -> bool correctness.

    Prints answers per position and the discriminating delta:
    accuracy(first) - accuracy(middle). A positive delta supports lost-in-the-middle.
    """
    results = {}
    for position in ("first", "middle", "last"):
        ordered = place(gold, distractors, position)
        context = "nn---nn".join(ordered)
        prompt = (
            f"Context:n{context}nn"
            f"Question: {query}nn"
            "Answer using only the context."
        )
        answer = ask(prompt)
        ok = judge(answer) if judge else None
        results[position] = {"answer": answer, "ok": ok}
        print(f"{position:6} ok={ok}  {answer[:120]!r}")

    if judge is not None:
        first = 1.0 if results["first"]["ok"] else 0.0
        middle = 1.0 if results["middle"]["ok"] else 0.0
        delta = first - middle
        print(f"accuracy(first)-accuracy(middle) = {delta:+.0f}")
        if delta > 0:
            print("CAUSE lost-in-the-middle -> U-shape reorder + cut top-k")
        elif not results["first"]["ok"] and not results["middle"]["ok"]:
            print("Gold placement did not help — re-check that `gold` truly answers the query")
        else:
            print("No middle penalty on this item — look at wrong-chunk / conflict / prompt")
    return results

def ushape_order(ranked_best_first):
    """Place strongest evidence on the edges; weakest in the middle.

    ranked_best_first: list already sorted by relevance descending (A, B, C, D, E).
    Example: [A, B, C, D, E] -> [A, C, E, D, B]
    """
    left, right = [], []
    for i, chunk in enumerate(ranked_best_first):
        (left if i % 2 == 0 else right).append(chunk)
    return left + list(reversed(right))

Wire ask to your production chat API and judge to a string match or a labelled expected answer. After the delta confirms the failure, call ushape_order on your reranked list before you build the prompt. Generation-side scoring after the fix belongs with generation metrics.