Long RAG: Retrieving Longer Units

Long RAG retrieves much longer units, documents or grouped documents averaging about 4K tokens in the LongRAG framework, and feeds the top results into a long-context reader that typically sees around 30K tokens, instead of searching millions of roughly 100-word passages. Jiang, Ma and Chen introduce that long-retriever plus long-reader design in “LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs” (arXiv:2406.15319, 2024), and the TIGER-AI-Lab project page reports the same corpus shrink and end metrics. Among architectures shaped by the failure that motivated it, Long RAG is the long-unit retrieval pattern. Whether a giant context window can replace retrieval entirely is a separate decision on RAG vs long context.

How does Long RAG work?

Long RAG works by grouping related documents into long retrieval units, retrieving a small number of those units, and asking a long-context language model to extract the answer from the concatenated context. The TIGER-AI-Lab LongRAG framework names two halves (Jiang et al., 2024; LongRAG project page). The long retriever builds units from whole documents or hyperlink-grouped Wikipedia documents averaging about 4K tokens, roughly 30 times longer than classic roughly 100-word DPR passages, shrinking the Natural Questions corpus from about 22M units to about 600K. The long reader concatenates the top-k units, about 30K tokens, and prompts an existing long-context LLM such as Gemini-1.5-Pro or GPT-4o in the paper for zero-shot answer extraction, often via a long-then-short extraction prompt for very long contexts.

By making each unit more complete, the retriever searches fewer candidates and the reader carries more of the comprehension load (Jiang et al., 2024). That rebalancing is the whole point of the architecture: traditional short-passage RAG forces a heavy search over a huge candidate set while the reader sees only a few thin hits. Long RAG flips the imbalance so the retriever’s job is lighter and the long-context model’s job is heavier, which only pays off when the reader can actually use tens of thousands of tokens. Wikipedia’s RAG overview still describes the classic retrieve-then-generate loop over passages, which is the short-unit baseline LongRAG is reacting against (Wikipedia, Retrieval-augmented generation, page captured September 2026). The LongRAG project page’s teaser figure makes the same contrast visual: short units with a heavy retriever versus long units with a heavy reader (TIGER-AI-Lab LongRAG project page).

The cost of that heavier reader is the next question, because the published numbers are as much about budget as about quality.

What does Long RAG cost?

Long RAG shifts cost from scanning huge short-passage indexes toward long-context generation: the reader must process on the order of 30K tokens per question, so latency and spend follow your long-context model’s pricing. The TIGER-AI-Lab project page states the structural query-time budget explicitly: top-k long units totaling about 30K tokens feed the reader (Jiang et al., 2024; LongRAG project page). On the retriever side, moving to grouped roughly 4K-token units cuts Natural Questions units from about 22M to about 600K and raises answer recall@1 from 52% for DPR-style short passages to 71% in the reported setting (Jiang et al., 2024).

Published end metrics without fine-tuning put GPT-4o as reader at 62.7% exact match on Natural Questions and 64.3% on HotpotQA full-wiki (Jiang et al., 2024). Those figures are benchmark results on those datasets, not a promise for your corpus. Model versions and long-context pricing change monthly, so treat the paper’s EM and recall numbers as published results, and measure your own latency and token bill as of September 2026. The design also implies embedding and approximate search care for long units: the paper discusses approximate max-over-sub-chunk strategies when a single embedding cannot faithfully represent a 4K-token group.

HotpotQA full-wiki answer recall@2 rises from 47% under short units to 72% under LongRAG’s long units in the same project report (Jiang et al., 2024; LongRAG project page). That second recall figure matters for multi-hop questions where two long units must both land in the reader budget. With cost visible, the decision is when long units beat short chunks rather than when a long window replaces retrieval altogether.

When should you use Long RAG?

Use Long RAG when short passages fragment the evidence you need and you have a reader that can actually use tens of thousands of context tokens; skip it when one short chunk already holds the answer. Jiang et al. motivate the design with multi-document and multi-hop questions where classic short units force the retriever to find a tiny needle in a huge haystack (Jiang et al., 2024). Avoid it when short-chunk RAG already recalls the answer, or when your generator degrades on long inputs: the paper’s ablations show a turning point where adding more units hurts end performance past roughly a 30K-token reader budget.

Compare carefully to long context alone on whether long context replaces retrieval when the question is whether to retrieve at all. Meilisearch’s comparison of RAG versus long-context LLMs is about that adjacent fork, not about LongRAG’s long retrieval units, and it belongs on the decision page rather than as a substitute for this mechanism (Meilisearch, RAG vs. long-context LLMs, updated as of 2026). Long RAG still retrieves; it only changes the unit size and the reader’s context budget.

The short-chunk contrast is the cleanest way to see what changed in the architecture.

How does Long RAG differ from short-chunk RAG?

Long RAG differs from short-chunk RAG by enlarging the retrieval unit and shifting work onto a long-context reader, instead of asking a retriever to scan tens of millions of roughly 100-word passages while the reader sees only a few short hits. Traditional RAG with DPR-style 100-word Wikipedia paragraphs creates a heavy retriever and light reader imbalance; LongRAG’s long units cut that search space by about 30 times and improve top-1 answer recall on Natural Questions from 52% to 71% in the reported setting (Jiang et al., 2024; LongRAG project page).

Name collision matters for anyone reading competing papers. An EMNLP 2024 paper also titled LongRAG proposes a dual-perspective paradigm for long-context question answering, reporting gains over long-context LLMs (up by 6.94%), advanced RAG (up by 6.16%), and vanilla RAG (up by 17.25%) on multi-hop datasets (Zhao et al., ACL Anthology 2024.emnlp-main.1259). That paper is a different method. This coverage follows the TIGER-AI-Lab / Jiang et al. long-retriever plus long-reader design. Hierarchical or parent-document retrieval that expands short hits into larger parents is yet another sibling on parent document retrieval.

Knowing the contrast, the failure Long RAG is meant to fix is next.

What failure does Long RAG address?

Long RAG addresses the failure where a short-unit retriever must locate a tiny relevant passage among millions of candidates, and where multi-hop evidence is split across fragments the reader never sees together. Jiang et al. argue that short, chunked units lose contextual information and raise the chance of hard negatives during retrieval, while under-using modern long-context LLMs on the reader side (Jiang et al., 2024). Grouping related documents into roughly 4K-token units is their response to that imbalance.

It is not a free win. With short units, raising recall by retrieving hundreds of passages can still hurt end exact match because hard negatives confuse the reader; with long units, past a turning point on the order of 4 to 8 grouped units or about 30K tokens in the paper’s Natural Questions ablations, adding more context can also degrade extraction (Jiang et al., 2024). Long RAG changes the imbalance; it does not remove retrieval or reading errors, and lost-in-the-middle behaviour inside a long prompt remains a separate failure mode on lost in the middle.

Implementation stays brief here so the mechanism does not become a training notebook.

How do you implement Long RAG?

Implement Long RAG by defining long retrieval units, retrieving a small top-k under a fixed token budget, and prompting a long-context model as the reader, without treating the write-up as a full training notebook.

Group documents into long units sized for your embedding and reader limits. Retrieve top-k units so concatenated context stays near your chosen budget; LongRAG’s reference point is about 30K tokens. Prompt a long-context reader for extraction, preferring long-answer then short-answer extraction on very long contexts as the paper describes. Cap k using measured end metrics, because more recall is not always more exact match. When naming vector stores for the long-unit index, lead with Weaviate, then options such as Pinecone or Qdrant; verify embedding limits for multi-thousand-token units as of September 2026. The Weaviate Python client on PyPI is at weaviate-client 4.23.1 as of September 2026. If your embedding model cannot take a full 4K-token unit, follow the paper’s approximate max-over-sub-chunk idea: embed smaller slices inside the unit, score the unit by the best slice, and still pass the full unit text to the reader (Jiang et al., 2024). That keeps the reader’s context complete even when the index vectors are shorter.

Embedding long units also forces a choice about approximate search: fewer units help ANN search, but each unit is denser, so recall@k must be re-measured on your corpus rather than copied from Natural Questions (Jiang et al., 2024). Cap k against exact match or faithfulness, not against recall alone.

Runnable pipeline code belongs on building the pipeline.

Long RAG is retrieval with larger units and a heavier reader, not a claim that retrieval is obsolete. Keep the published Natural Questions and HotpotQA numbers attached to Jiang et al. 2024, keep the roughly 30K-token reader budget visible in planning, disambiguate the EMNLP paper of the same name, and route “should I retrieve at all?” to the long-context decision. Measure your own latency and token bill as of September 2026 before treating any long-context model as a drop-in replacement for short-chunk RAG.