FLARE: Forward-Looking Active Retrieval

FLARE (Forward-Looking Active REtrieval augmented generation) is an active RAG method that predicts the upcoming sentence, retrieves when that draft contains low-confidence tokens, and regenerates the sentence with the new documents — instead of retrieving once from the user query only. Jiang et al. (2023; EMNLP 2023; arXiv:2305.06983) introduce FLARE as a generic inference-time method for long-form knowledge-intensive generation; official code is at jzbjyb/FLARE. This page covers the mid-generation loop, how queries are formed, the published Table 1–2 numbers, and when the paper says FLARE does not help.

How does FLARE decide when to retrieve mid-generation?

FLARE decides mid-generation by drafting the next sentence first, then retrieving only if that draft looks under-confident — a forward-looking gate, not a fixed retrieve-every-N-tokens schedule.

1. Draft the next sentence. Conditioned on the user input and text already accepted, the LM generates a temporary next sentence (Jiang et al., 2023, Figure 1 / §3.2).
2. Check token probabilities against θ. If every token’s probability is ≥ θ, keep the draft. If any token falls below θ, trigger retrieval. θ = 0 never retrieves; θ = 1 retrieves every sentence (paper §3.2).
3. Build a query from the draft. Mask low-confidence tokens or generate explicit search questions (next section).
4. Retrieve and regenerate. Fetch documents (BM25 over Wikipedia, or Bing for WikiAsp in the paper’s setup), prepend them, and regenerate the sentence.
5. Repeat until the answer ends. Each accepted sentence becomes context for the next draft.

The paper’s default reported system is FLARE_direct (confidence-triggered). FLARE_instruct instead teaches the model to emit [Search(query)] tokens (§3.1); on 2WikiMultihopQA Table 1 it scores lower than Direct (EM 42.4 vs 51.0).

How does FLARE turn a low-confidence sentence into a retrieval query?

FLARE forms the retrieval query from the temporary sentence so the search matches what the model is about to say, not only the original user question. Jiang et al. (2023 §3.2.2) give two formulations:

• Implicit (masking). Replace tokens with probability below β with masks so erroneous low-confidence spans do not dominate the query. Higher β masks more aggressively. Table 5 shows β = 0 (full sentence) underperforms mild masking on 2WikiMultihopQA.
• Explicit (question generation). Ask the LM to turn uncertain spans into natural-language search questions. Table 6 finds implicit and explicit performance similar on ASQA-hint and WikiAsp.

Paper Table 9 (dev-tuned) sets β = 0.4 on all four datasets; θ = 0.8 for 2WikiMultihopQA, ASQA/ASQA-hint, and WikiAsp, and θ = 0.4 for StrategyQA. The paper’s running example is why masking matters: a draft that wrongly sends Joe Biden to the University of Pennsylvania can retrieve the wrong school unless uncertain tokens are stripped or re-queried before search.

How well does FLARE perform on published benchmarks?

FLARE’s published gains are from Jiang et al. (2023) few-shot runs with text-davinci-003, at most 500 examples per dataset, BM25 or Bing retrieval as listed in the paper’s Table 7 — not a guarantee on your corpus.

On 2WikiMultihopQA, FLARE_direct’s EM of 51.0 beats question-decomposition’s 47.8 and single-time’s 39.4 (Table 1). Ablations in the same paper show next-sentence queries beat previous-sentence queries (Table 3), and that retrieving on roughly 40–80% of sentences is usually a good operating range — past 50% retrieval on StrategyQA can hurt (Figure 5 discussion).

What does FLARE cost?

FLARE’s cost is structural multi-call generation: each sentence may require a draft call plus a regenerate call after retrieval, so API and latency scale with output length and how often θ fires.

• Multiple LM activations per example. The official FLARE repository warns that experiments are relatively expensive because FLARE calls the OpenAI API multiple times for a single example (jzbjyb/FLARE README).
• Engineering overhead. Jiang et al. (2023 §9) note that a naive interleaving implementation increases overhead and cost: the LM is activated once per retrieval step, and a caching-free stack recomputes prior activations after each retrieve.
• No universal dollar figure. Exact spend depends on model price, sentence count, and retrieval rate — measure on your stack; do not invent a latency SLA from the paper.

When should you use FLARE, and when should you avoid it?

FLARE is built for long-form generation where later sentences need facts the initial retrieve never saw. Jiang et al. (2023 §9) also document settings where it did not help.

• Use it for long, knowledge-intensive outputs. Multihop QA, long-form QA, and open-domain aspect summarization are the paper’s positive tasks (2WikiMultihopQA, ASQA, WikiAsp, StrategyQA).
• Expect little gain on short dialogue turns. On Wizard of Wikipedia, outputs average about 20 tokens — the paper reports no significant FLARE gains (§9).
• Expect little gain on ELI5-style open-ended long answers. The paper likewise found no significant gains for FLARE or single-time retrieval over no retrieval on ELI5, citing grounding and evaluation difficulties (§9).
• Skip it if you lack token probabilities or multi-call budget. The Direct gate needs per-token confidence; without that signal, you cannot implement paper-faithful θ triggering.

If you only need a once-per-query skip/retrieve decision, that is adaptive retrieval, not FLARE.

How is FLARE different from adaptive retrieval or IRCoT?

FLARE is one active-retrieval design among several that share the “retrieve more than once” slogan but trigger on different events.

• Adaptive retrieval gates whether to call the retriever for the query at all (classifier, confidence, or Retrieve-token). It is usually decided before or as the answer starts — /retrieval/adaptive.
• IRCoT interleaves retrieval with chain-of-thought steps so each thought can fetch what the next needs — IRCoT.
• Single-time RAG retrieves once from the user input, then generates the whole answer.
• Fixed-interval multi-time retrieval retrieves every N tokens or every sentence without checking confidence — the passive baselines FLARE beats in Jiang et al. (2023 §4 / Tables 1–2).

General multi-hop control-loop depth lives on multi-hop RAG.

How do you implement FLARE?

Implementing FLARE is a handoff checklist: expose token probabilities, loop sentence drafts, apply θ/β, retrieve, regenerate, and re-tune thresholds when you change models.

1. Require logprobs (or equivalent confidence). Direct FLARE is undefined without them.
2. Loop sentence generation. Draft → gate on θ → optional retrieve → accept regenerated sentence.
3. Wire a retriever. The paper uses BM25 on a Wikipedia dump or Bing; production stacks swap in a vector index (Weaviate and other stores appear in RAG tooling roundups) behind the same query step.
4. Start from published configs, then re-measure. Table 9’s θ/β values were tuned for text-davinci-003 (paper notes API access as of April 2023) — re-tune on a modern model.

Official code is jzbjyb/FLARE; LangChain / DataStax / LanceDB tutorials exist as examples. Runnable pipeline assembly belongs on building the pipeline.