Lost in the Middle: When the Right Chunk Is Ignored

Lost in the middle is the failure where the gold chunk already sits in the assembled prompt and the model still under-uses it because of position. Retrieval succeeded; generation treated the middle of a long context as low-salience. That is not a wrong chunk, where the answer-bearing text never entered the retrieved set, not a missing document, where nothing relevant was a candidate at any k, and not conflicting sources, where two passages disagree on the fact. Confirm the gold passage is inside the prompt before you change the retriever. The leaf sits in the failure taxonomy sorted by the stage that caused it: this one is generation over a context that retrieval already filled, and the usual fix is how you order and truncate that context rather than a new embedding model.

How do you know it is lost in the middle?

You know it is lost in the middle when the gold chunk is present in the prompt and moving that same chunk from the middle to the start or the end measurably improves the answer. Other failures look identical from the outside: the user sees a confident wrong answer after a retrieval step that looked healthy. The discriminating move is one position swap on the same gold text before you rewrite prompt assembly or replace the vector store.

Atlan’s June 2026 guide on the lost-in-the-middle problem (Emily Winks) states the same operational test without naming sibling failure nodes: take one fact the model should answer correctly, place it near the beginning, in the middle, and near the end, ask the same question each time, and compare. DEV Community’s thousandmiles walkthrough (captured September 2026) frames the production scene as five relevant chunks with the answer in chunk three and a model that still leans on the edges. Neither page publishes a three-way table that routes the reader to wrong-chunk or missing-document when the gold text is absent. That table is the gate here.

Debugging guides often jump straight to reorder your chunks. That only helps when the position delta is real. If the gold passage is not in the prompt at all, reordering cannot recover it; leave this leaf and open the matching failure. For the full stage walk across retrieval and generation, see how to find which stage of your RAG pipeline broke. Once the delta is positive on the same gold chunk, the next question is what the literature actually measured about that middle position.

What did Liu et al. measure about context position?

Liu et al. measured that language models do not use long input contexts robustly: performance is often highest when the relevant information sits at the beginning or the end of the context, and it degrades significantly when the same information sits in the middle, including for models marketed as long-context. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni and Percy Liang published that result as Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172, revised November 2023; accepted in Transactions of the Association for Computational Linguistics). The abstract states the two tasks plainly: multi-document question answering and key-value retrieval.

Maxim’s production article on solving the lost-in-the-middle problem (captured September 2026) summarises the study’s context lengths as settings of 10, 20 and 30 documents and describes the resulting accuracy curve as U-shaped: edges high, middle low. That 10/20/30 framing is Maxim’s paraphrase of the paper’s protocol, not a new measurement. AI Planet’s LongContextRetriever walkthrough (Plaban Nayak) states that seven language models, open and closed, including GPT-3.5 16K and Claude 1.3 with a 100K window, displayed the same U-curve pattern when the solution sat at the beginning or end rather than the middle. Those named model families and window sizes are AI Planet’s secondary report of the study; they are usable as such, dated to that write-up.

What this session cannot cite is any closed-book or oracle percentage from the paper’s tables. The Stanford PDF ranked in the organic set returned zero extractable characters, and the arXiv abstract page does not publish Table figures. Maxim’s claim that performance can degrade by more than 30 percent is a secondary paraphrase that is not verified against the paper text here, so it is not repeated as Liu’s number. Blog reconstructions that invent a middle-position percentage such as roughly 65 percent are likewise refused. The behavioural claim that stands is the U-shaped position effect on multi-document QA and key-value retrieval, measured by moving the single answer-bearing document through the context while holding length fixed.

If a pipeline concatenates top-k chunks in rank order, chunk one lands at the start and the mid-ranked hits drift into the middle, exactly the region the U-curve under-uses. Relevance at retrieval time is not the same as visibility at generation time. The mechanism write-up of ordering lives on how prompt position changes whether a fact is used. The behavioural curve still leaves open why decoder-only models lean on the edges in the first place.

Why do language models ignore the middle of the prompt?

Language models ignore the middle of the prompt because decoder-only transformers exhibit primacy bias on early tokens and recency bias on late tokens, so middle positions receive less effective use even when every token is technically inside the window. Liu et al. document the behavioural U-curve; follow-up practitioner write-ups attribute the pattern to how causal attention and positional encodings weight sequence positions.

thousandmiles’ DEV Community explanation (captured September 2026) names two architectural contributors without shipping new benchmark percentages. Causal attention masking lets each token attend only to earlier tokens, so early positions accumulate more opportunities to be attended across the sequence. Rotary position encodings introduce distance-based decay, so tokens far from both the start and the generation end sit in a weaker attention band. Maxim’s article likewise points at RoPE long-term decay as a common explanation in the follow-on literature. Those are attributions to architecture research summarised by practitioners, not new measurements invented here.

This leaf stays on the failure you can measure in a RAG prompt. Architectural mitigations such as multi-scale positional encodings, attention calibration, and specialised information-intensive training belong on the ordering mechanism page, not here. Embedding choice did not cause this failure: embeddings decide what gets retrieved into a store such as Weaviate, Pinecone, Qdrant or Milvus; position decides what gets attended to once those passages are already in the prompt. Serial-position effects in human list memory are a useful parallel Liu and later writers note, not a proof that the model “forgets” like a person. Once the why is framed as position rather than missing retrieval, the next myth to kill is that a bigger advertised window removes the blind spot.

Does a longer context window fix lost in the middle?

A longer context window does not fix lost in the middle. Liu et al.’s abstract states that the middle degradation appears even for explicitly long-context models. Extending the window mostly adds more middle for distractors and mid-ranked chunks to occupy, rather than equalising attention across every token.

thousandmiles records the practical myth in one line: the 2023 paper already showed the U-curve across models with 4K, 16K and 32K windows, and later work still finds the pattern in 128K-class windows. Atlan’s guide (updated July 2026) cites Chroma’s 2025 context-rot report, which tested 18 LLMs including GPT-4.1, Claude 4, Gemini 2.5 and Qwen3 families and found that newer models still do not use context uniformly as input length grows. The same Atlan piece cites research on Maximum Effective Context Window (arXiv 2509.21361) for the claim that, in that paper’s tests, every evaluated model fell short of its advertised maximum by as much as 99 percent. Those figures belong to Chroma and to the MECW authors as Atlan reports them; they corroborate usable-context limits, they are not Liu Table numbers.

Atlan also maps the enterprise shape of the same failure: a RAG system retrieves the right chunk, then buries it among weaker neighbours; a BI assistant parks a certified metric definition mid-prompt while a legacy SQL snippet sits nearer the question; an agent session accumulates tool output until a policy rule is no longer salient. In each case the evidence was present and still under-used. The practical control on the retrieval side is therefore cutting how many chunks you keep, not the marketing maximum on the model card. Hub depth for retrieval design sits under retrieval design. Once longer windows are ruled out as a cure, the production moves are reorder, cut k, and only then prompt tricks.

How do you fix lost in the middle in a RAG prompt?

You fix lost in the middle by placing the highest-relevance chunks at the start and end of the context, putting weaker chunks in the middle, and cutting top-k until extra documents stop helping the answer. Fighting the U-curve is less reliable than working with it. Reranking alone does not fix the failure unless the scores change which chunk lands on the edges after you reorder.

Four production moves, in the order most teams should try them, match the union of thousandmiles, Maxim, Medium’s LongContextReorder posts, AI Planet’s reorder pipeline, and Sewak’s LinkedIn toolkit of placement, reranking and compression.

1. U-shape the order after you score relevance. Score candidates with a cross-encoder reranker, then place the best chunk first, the second-best last, and the weakest in the middle. For five chunks ranked A≻B≻C≻D≻E, a typical edge-first layout is A, C, E, D, B: A and B on the edges, E in the centre. thousandmiles uses a related pattern of best at start, second-best at end, least important in the middle. LangChain’s LongContextReorder, named in Juan C. Olamendy’s Medium deep dive and in AI Planet’s LongContextRetriever pipeline, implements the same idea; the algorithm matters more than the framework. Full ordering mechanics: where to put the best chunk in the prompt.
2. Cut top-k. Maxim’s production guidance is to retrieve generously for recall, then keep only the most relevant 3 to 5 documents for generation. thousandmiles gives the same surgical range after a reranker. Prefer a short, edge-heavy window over a long middle. Tune k on how many chunks should you retrieve.
3. Query-aware contextualization. Place the user question both before and after the documents so the model can condition on the query while reading. Liu et al. study query placement as part of how models use long contexts; secondary summaries report stronger effects on key-value style probes than on multi-document QA. Treat the dual-query pattern as a cheap experiment on your own labelled set, not as a substitute for reorder and cut-k, and do not invent key-value accuracy percentages the abstract does not publish.
4. An explicit “use every passage, including the middle” instruction can help at the margin, as thousandmiles notes. It is not a substitute for reorder and cut-k. Prompt compression and multi-pass extraction are valid deeper mitigations; depth for those lives on prompt assembly order and on RAG prompts.

Hybrid search, new embeddings, and chunk-size retunes fix cases where the gold text never entered the window. Those belong with retrieval returning the wrong passage, not with position bias after a successful retrieve. Once the production order is clear, the last job on this leaf is to measure the position delta on your own model before you ship the reorder.

How do you run a position test on your own pipeline?

You run a position test by taking one failing query, the chunk you know contains the answer, and a handful of distractors, then asking the same model three times with the gold chunk at the first, middle and last positions. A middle miss with a first or last hit is the lost-in-the-middle signature. Atlan’s middle-position failure-rate test is the same geometry applied to the context your system actually ships: retrieved chunks, metric definitions, policies or tool instructions.

Wire the helper below to your production chat API and to a string match or labelled expected answer. One anecdotal flip is a hint, not a proof. Run the swap across a small labelled set of failing queries on the model you actually serve, then treat a positive accuracy(first) − accuracy(middle) as the gate that authorises U-shape reorder and a smaller k. Generation-side scoring after the fix belongs with generation metrics; if the first-position answer is still wrong, the fault has left this leaf.

def place(gold, distractors, position):
    """Return chunks with `gold` at first | middle | last among distractors."""
    others = list(distractors)
    if position == "first":
        return [gold] + others
    if position == "last":
        return others + [gold]
    mid = len(others) // 2
    return others[:mid] + [gold] + others[mid:]

def position_test(query, gold, distractors, ask, *, judge=None):
    """ask(prompt) -> answer string. Optional judge(answer) -> bool correctness.

    Prints answers per position and the discriminating delta:
    accuracy(first) - accuracy(middle). A positive delta supports lost-in-the-middle.
    """
    results = {}
    for position in ("first", "middle", "last"):
        ordered = place(gold, distractors, position)
        context = "\n\n---\n\n".join(ordered)
        prompt = (
            f"Context:\n{context}\n\n"
            f"Question: {query}\n\n"
            "Answer using only the context."
        )
        answer = ask(prompt)
        ok = judge(answer) if judge else None
        results[position] = {"answer": answer, "ok": ok}
        print(f"{position:6} ok={ok}  {answer[:120]!r}")

    if judge is not None:
        first = 1.0 if results["first"]["ok"] else 0.0
        middle = 1.0 if results["middle"]["ok"] else 0.0
        delta = first - middle
        print(f"accuracy(first)-accuracy(middle) = {delta:+.0f}")
        if delta > 0:
            print("CAUSE lost-in-the-middle -> U-shape reorder + cut top-k")
        elif not results["first"]["ok"] and not results["middle"]["ok"]:
            print("Gold placement did not help; re-check that gold truly answers the query")
        else:
            print("No middle penalty on this item; look at wrong-chunk / conflict / prompt")
    return results

def ushape_order(ranked_best_first):
    """Place strongest evidence on the edges; weakest in the middle.

    ranked_best_first: list already sorted by relevance descending (A, B, C, D, E).
    Example: [A, B, C, D, E] -> [A, C, E, D, B]
    """
    left, right = [], []
    for i, chunk in enumerate(ranked_best_first):
        (left if i % 2 == 0 else right).append(chunk)
    return left + list(reversed(right))

After the delta confirms the failure, call ushape_order on your reranked list before you build the prompt. Store the three-position outcomes beside the query id the way you would store any other regression case. If first and middle both fail, the gold chunk may not answer the question, or retrieval never had the right text, and you are back on the differential table rather than on reorder.

Lost in the middle is a position failure after retrieval already succeeded: the gold chunk is in the prompt, the model under-uses the middle, and the discriminating proof is a positive accuracy gap when that same chunk moves to the edge. Liu et al. established the U-shaped behavioural pattern on multi-document QA and key-value retrieval; longer windows do not erase it; production systems respond by scoring relevance, parking the best evidence on the edges, cutting k, and measuring the swap on the model they ship. When the gold text was never retrieved, leave this leaf. When it was retrieved and ignored in the middle, reorder and shorten before you blame the generator’s priors.