Where to Put the Best Chunk in the Prompt

Context ordering places scored retrieved chunks so the highest-relevance evidence sits where the model attends — typically the start and end of the packed prompt — because mid-context under-use can ignore a gold passage that retrieval already returned. It is the placement step inside context assembly, and the mechanism twin of lost in the middle.

Why does chunk order matter in a RAG prompt?

Chunk order matters because decoder-only language models use early and late context more reliably than the middle, so concatenating chunks in plain rank order parks mid-ranked evidence in the worst band of the prompt. Liu et al. (2023; TACL 2024) measured a U-shaped multi-document QA accuracy curve: models do best when the answer-bearing document sits at the beginning or the end, and worse when it sits in the middle — GPT-3.5-Turbo can drop by more than 20 percentage points on that move. Relevance at retrieval time is not the same as visibility at generation time. The full published table, the discriminating position test, and the differential against wrong-chunk / missing-document live on lost in the middle; this page owns how you place chunks once you know position is the failure.

How do you reorder chunks to mitigate lost in the middle?

You reorder after you have relevance scores: put the best chunk first, the second-best last, and the weakest in the middle — a U-shape that works with primacy and recency instead of fighting them. LangChain’s community LongContextReorder implements the same idea; the algorithm matters more than the framework.

For five chunks already ranked best-first as A ≻ B ≻ C ≻ D ≻ E, the edge-first layout is A, C, E, D, B — A and B on the edges, E in the centre. One equivalent construction: alternate chunks into a left list and a right list, then return left concatenated with reversed(right). LangChain’s documented _litm_reordering (reverse the best-first list, then insert on even indices and append on odd — as listed by Jagat, Level Up Coding, September 2025) yields the same A, C, E, D, B order. Juan C. Olamendy’s October 2023 walkthrough is the early popularisation of that transformer for RAG prompts.

1. Score the shortlist. Prefer a cross-encoder reranker over raw bi-encoder rank when you can afford the latency; otherwise keep retrieval scores.
2. Sort best-first. The U-shape assumes the input list is already relevance-descending.
3. Place on the edges. Apply the alternate-edges reorder before you concatenate into the prompt.
4. Respect the token budget. Ordering does not invent tokens — if the packed context still overflows, cut or compress after you place, not before you know what must sit on an edge.

Should you put the question before or after the retrieved documents?

Prefer query-aware placement: put the user question where the model can condition on it while reading the evidence. Liu et al. (§4.2) found that placing the question both before and after the documents made key-value retrieval near-perfect — GPT-3.5-Turbo (16K) reached perfect accuracy at 300 key-value pairs — while without it the worst-case key-value accuracy was 45.6%. On multi-document QA the same trick barely moved the U-curve; state that honestly when you adopt it. Prompt-design write-ups such as The GenAI Revolution’s lost-in-the-middle note also recommend leading with the question so the attention path to evidence is shorter. Template depth belongs on RAG prompts.

Does context order still matter with newer long-context models?

Context order still matters for many production models: Liu et al. found that when the same 10- or 20-document inputs fit both a base model and its extended-context counterpart (for example GPT-3.5-Turbo versus GPT-3.5-Turbo 16K), the position curves were nearly superimposed — a bigger window mostly adds more middle. As of September 2025, Jagat’s Level Up Coding replication on the first 100 rows of the bzantium/LITM qa20 set reported a flatter curve for GPT-4.1-nano than for GPT-3.5-turbo; that is a blog experiment on one checkpoint and one slice, not a proof that reordering is obsolete. Thousandmiles’ 2026 summary is blunt that no production model has fully eliminated position bias. The rule: run the position test on the model you serve before you delete the reorderer — and do not confuse a larger advertised window with better mid-context use (RAG vs long context).

What architectural changes reduce middle-context loss?

Architectural changes can reduce middle-context loss, but prompt-side U-shape ordering remains the production default because it needs no model fork. Multi-scale Positional Encoding (Ms-PoE) — Zhang et al., Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding (arXiv:2403.04797) — rescales RoPE position indices per attention head (default published range 1.2–1.8) without fine-tuning. The paper’s reported gains are an average accuracy improvement of up to 3.8 on ZeroSCROLLS over the original LLMs, and a reduction of the best-versus-worst position gap by roughly 2–4 percentage points, with additional sweet-region gains in the 3–6% range on their multi-document and key-value setups (§4). Do not treat vendor-blog paraphrases that invent a “20–40% middle-position accuracy” lift as Zhang et al.’s numbers — that figure is not in the paper’s abstract or §4 results.

Training-time approaches such as Information-Intensive (IN2) training and models in the FILM line (Microsoft Research, as summarised in industry write-ups including getmaxim) teach the model with relevant facts placed at varied positions so attention is less boundary-locked. Published FILM percentage lifts are not restated here without opening those primary sources — verify before you rely. Attention-calibration and multi-pass inference appear in the same literature; they cost more than a deterministic reorder. Detection and the symptom walk stay on lost in the middle; packing still starts at context assembly.

When is reordering not enough?

Reordering is not enough when the gold chunk never entered the prompt, or when top-k is so large that even an edge-heavy layout still leaves a long, noisy middle. Next moves, in order most teams should try:

• Cut top-k. Liu et al. saw only about 1.5 percentage points (GPT-3.5-Turbo) and about 1 percentage point (Claude-1.3) of reader gain past 20 retrieved documents while recall was still rising. Tune k on choosing top-k.
• Compress mid-ranked text. Extractive or abstractive shrink keeps claims under budget without inventing a longer middle — see contextual compression.
• Multi-pass extraction or an explicit middle-attention instruction. Process documents independently then synthesise, or tell the model to use every passage including the middle. Both help at the margin; neither replaces reorder and cut-k. Prompt patterns: RAG prompts.

If the gold text is absent from the assembled prompt entirely, leave this page — wrong chunk or missing document is the failure class.

How do you implement context ordering?

You implement context ordering by sorting the retrieved (or reranked) shortlist best-first, running the U-shape reorder, then concatenating under your token budget — LangChain Community ships LongContextReorder for that transform, and any stack can implement the same alternate-edges function without a framework lock-in. Pin library versions when you paste framework code into building the pipeline. Measure before/after with the position test already shown on lost in the middle; generation-side scoring after the change belongs with generation metrics.