IRCoT: Interleaving Retrieval with Chain-of-Thought

IRCoT (Interleaved Retrieval guided by Chain-of-Thought) is the Trivedi et al. (ACL 2023; arXiv:2212.10509) retrieve-and-read method that alternates chain-of-thought sentence generation with retrieval, so each reasoning step can fetch the evidence the next step needs. Official code ships at github.com/StonyBrookNLP/ircot. This page covers the interleave loop, the stop rule, the published gains, and the per-iteration cost that one-shot RAG does not pay.

How does IRCoT interleave retrieval with chain-of-thought?

IRCoT turns retrieval into a loop guided by CoT sentences rather than a single question query. Trivedi, Balasubramanian, Khot and Sabharwal (2023, §3.1 and Figure 2) describe three phases:

1. Initial retrieval. Retrieve K paragraphs from the corpus using the user question as the query.
2. Reason step. Generate the next CoT sentence from the question, the paragraphs collected so far, and the CoT sentences generated so far.
3. Retrieve step. Use that latest CoT sentence as the new query, retrieve K more paragraphs, and add them to the collection.

The loop continues until a termination rule fires. The paper’s motivating failure is one-shot retrieve-and-read: for a multi-step question such as where Lost Gravity was manufactured, the Wikipedia page returned by the question alone may never mention the manufacturing country — the system must first derive the manufacturer (Mack Rides) and only then retrieve with that entity (Trivedi et al., 2023, Introduction). That is the same surface as iterative multi-hop RAG; IRCoT’s specific move is to let each CoT sentence be the next retrieval query. Related sub-query splitting without the CoT-as-query loop sits at query decomposition.

When does the IRCoT loop stop?

IRCoT stops when either of two conditions holds (Trivedi et al., 2023, §3.1): the generated CoT contains the string answer is:, or the maximum number of reasoning steps is reached. The paper sets that maximum to 8 steps in its experiments. After termination, the full collected paragraph set is returned as the retrieval result and passed to a QA reader that uses either direct prompting or CoT prompting (Trivedi et al., 2023, §3.2). AutoRAG-Research’s IRCoT pipeline docs (as of the page captured 2026-07-28) document the same stop string and a default max_steps of 8.

What results did the IRCoT paper publish?

The published gains are from the ACL 2023 paper’s own measurements — not a production SLA. Using GPT-3 (code-davinci-002), IRCoT improved retrieval by up to 21 points and downstream QA by up to 15 points versus one-step question-based retrieval on HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC (Trivedi et al., 2023, abstract). The body also reports an 11–21 recall-point gain under a fixed-budget optimal-recall setup (Trivedi et al., 2023, §5). On a manual CoT factuality check of 40 questions per dataset, IRCoT reduced factual errors over one-step retrieval (OneR) by 50% on HotpotQA and 40% on 2WikiMultihopQA (Trivedi et al., 2023, §5). The paper reports similar trends with Flan-T5 models without additional training, and gains that hold in out-of-distribution demonstration settings. How those datasets sit in a broader eval map is at RAG benchmarks.

What does IRCoT cost per iteration?

IRCoT’s cost is structural: each CoT sentence is a separate language-model call, so call count scales with reasoning length rather than with a single generate pass (Trivedi et al., 2023, Limitations). Every retrieve step also grows the context the next reason step must read — the authors note the need for long-context models (their GPT-3 setup allowed 8K tokens; Flan-T5 used relative position embeddings). A practitioner control documented by AutoRAG-Research (page captured 2026-07-28), not as a paper hyperparameter table, is paragraph_budget (example default 15) with a FIFO drop of earlier paragraphs so the collection cannot grow without bound. Exact dollars-per-query for a private corpus are not published — measure on your own stack. Broader pipeline spend framing is at pipeline cost.

When should you use IRCoT, and when should you avoid it?

IRCoT earns its multi-call overhead when later retrieval genuinely depends on entities derived mid-reasoning. It is usually the wrong default when one-shot retrieve-and-read already returns the supporting paragraphs.

• Use IRCoT for multi-step evidence chains. Questions where the second (or later) fact is not lexically available from the original question — the paper’s open-domain multi-step setting (Trivedi et al., 2023, Introduction).
• Avoid IRCoT for single-fact lookups. AutoRAG-Research’s when-to-use guidance (captured 2026-07-28) routes simple factual queries to basic one-shot RAG because IRCoT makes multiple LM calls; Medium’s hybrid note makes the same point (Tank, 2025).
• Respect the paper’s adoption limits. IRCoT assumes a base LM with zero- or few-shot CoT ability and enough context for many paragraphs; the authors also note that code-davinci-002 was deprecated after submission, while Flan-T5 trends remain reproducible from public weights (Trivedi et al., 2023, Limitations).

If you need a classifier that chooses zero-retrieval, single-step, or multi-step (IRCoT) per query, that routing architecture is Adaptive-RAG, not this page. The one-shot baseline is Naive RAG.

How does IRCoT differ from ReAct and multi-hop RAG?

IRCoT, ReAct and multi-hop RAG all iterate, but they do not iterate for the same reason. IRCoT interleaves retrieval with CoT sentences used as retrieval queries (Trivedi et al., 2023, §3.1). ReAct frames a Thought–Action–Observation loop where actions can be more general tool calls; the IRCoT paper contrasts that line in related work and notes ReAct’s original multi-step ODQA setup relied on much larger models and fine-tuning to beat CoT (Trivedi et al., 2023, §2, citing Yao et al., 2022). Multi-hop RAG is the broader iterative retrieve-and-compose family; IRCoT is one CoT-guided instantiation of that family. Adaptive-RAG routes some queries into an IRCoT strategy — the router itself is a different mechanism.

Depth lives on the siblings: ReAct RAG, multi-hop RAG, and Adaptive-RAG. Hop-miss versus composition-miss diagnosis stays on multi-hop failure diagnosis.

How do you implement IRCoT?

Implementing IRCoT is a handoff checklist, not a full reproduction notebook. Wrap a base retriever (the paper’s experiments use BM25) and a CoT-capable language model; run reason → retrieve with a max-step budget and an answer is: stop check; pass the collected paragraphs to a direct or CoT reader (Trivedi et al., 2023, §3.1–3.2).

1. Start from the official repo when you need paper fidelity. StonyBrookNLP/ircot is the ACL 2023 companion release.
2. Cap the loop. Set max steps (paper experiments use 8) and enforce the answer-string stop so the loop cannot spin.
3. Bound collected context. Cap total paragraphs (AutoRAG-Research documents a paragraph_budget with FIFO eviction) so later reason steps stay inside the model window.
4. Keep runnable pipeline code elsewhere. End-to-end build steps belong at building a RAG pipeline.