Long RAG: Retrieving Longer Units

Long RAG retrieves much longer units — documents or grouped documents, about 4K tokens on average in the LongRAG framework — and feeds the top results into a long-context reader that typically sees around 30K tokens, instead of searching millions of ~100-word passages (Jiang et al., arXiv:2406.15319, 2024).

In the architectures map, Long RAG is the long-unit retrieval pattern. Whether a giant context window can replace retrieval entirely is a separate decision on RAG vs long context.

How does Long RAG work?

Long RAG works by grouping related documents into long retrieval units, retrieving a small number of those units, and asking a long-context language model to extract the answer from the concatenated context.

The TIGER-AI-Lab LongRAG framework names two halves (Jiang et al., 2024; LongRAG project page):

1. Long retriever. Build units from whole documents or hyperlink-grouped Wikipedia documents averaging about 4K tokens — roughly 30× longer than classic ~100-word DPR passages — shrinking the NQ corpus from about 22M units to about 600K.
2. Long reader. Concatenate the top-k units (about 30K tokens) and prompt an existing long-context LLM (for example Gemini-1.5-Pro or GPT-4o in the paper) for zero-shot answer extraction, often via a long-then-short extraction prompt for very long contexts.

By making each unit more complete, the retriever searches fewer candidates and the reader carries more of the comprehension load (Jiang et al., 2024).

What does Long RAG cost?

Long RAG shifts cost from scanning huge short-passage indexes toward long-context generation: the reader must process on the order of 30K tokens per question, so latency and spend follow your long-context model’s pricing.

• Reader context budget ~30K tokens. LongRAG feeds top-k long units totaling about 30K tokens into the reader; that is the structural query-time cost (Jiang et al., 2024; LongRAG project page).
• Retriever corpus shrinks. On Natural Questions, moving to grouped ~4K-token units cuts units from about 22M to about 600K and raises answer recall@1 from 52% (DPR-style short passages) to 71% (Jiang et al., 2024).
• Published end metrics (no fine-tuning). With GPT-4o as reader, LongRAG reports 62.7% exact match on NQ and 64.3% on HotpotQA full-wiki (Jiang et al., 2024).

When should you use Long RAG?

Use Long RAG when short passages fragment the evidence you need and you have a reader that can actually use tens of thousands of context tokens; skip it when one short chunk already holds the answer.

• Use it for multi-document or multi-hop questions where classic short units force the retriever to find a tiny “needle” in a huge haystack (Jiang et al., 2024).
• Avoid it when short-chunk RAG already recalls the answer or when your generator degrades on long inputs — the paper’s ablations show a turning point where adding more units hurts end performance past roughly a 30K-token reader budget.
• Compare honestly to long context alone on RAG vs long context when the question is whether to retrieve at all.

How does Long RAG differ from short-chunk RAG?

Long RAG differs from short-chunk RAG by enlarging the retrieval unit and shifting work onto a long-context reader, instead of asking a retriever to scan tens of millions of ~100-word passages while the reader sees only a few short hits.

Traditional RAG with DPR-style 100-word Wikipedia paragraphs creates a heavy retriever / light reader imbalance; LongRAG’s long units cut that search space by about 30× and improve top-1 answer recall on NQ from 52% to 71% in the reported setting (Jiang et al., 2024; LongRAG project page).

Name collision: an EMNLP 2024 paper also titled “LongRAG” (dual-perspective paradigm) is a different method. This page follows the TIGER-AI-Lab / Jiang et al. long-retriever + long-reader design.

What failure does Long RAG address?

Long RAG addresses the failure where a short-unit retriever must locate a tiny relevant passage among millions of candidates, and where multi-hop evidence is split across fragments the reader never sees together.

It is not a free win. With short units, raising recall by retrieving hundreds of passages can still hurt end exact match because hard negatives confuse the reader; with long units, past a turning point (on the order of 4–8 grouped units / ~30K tokens in the paper’s NQ ablations) adding more context can also degrade extraction (Jiang et al., 2024). Long RAG changes the imbalance; it does not remove retrieval or reading errors.

How do you implement Long RAG?

Implement Long RAG by defining long retrieval units, retrieving a small top-k under a fixed token budget, and prompting a long-context model as the reader — without treating this page as a full training notebook.

1. Group documents into long units (whole docs or related-doc groups) sized for your embedding and reader limits.
2. Retrieve top-k units so concatenated context stays near your chosen budget (LongRAG’s reference point is about 30K tokens).
3. Prompt a long-context reader for extraction; for very long contexts, the paper prefers long-answer then short-answer extraction over one-shot span extraction.
4. Cap k using measured end metrics — more recall is not always more EM.

Runnable pipeline code belongs on building the pipeline. Verify long-context model APIs as of July 2026.