Query Rewriting and Expansion for RAG

Query rewriting reformulates the user’s question into one or more retrieval-friendly queries before search, clarifying, expanding, paraphrasing, or splitting it so the retriever matches how answers are phrased in the corpus. Expansion is the add-terms subset of that family; rewriting is the broader transform. The pipeline shape is rewrite-retrieve-read (Ma et al., EMNLP 2023, arXiv:2305.14283), not naive retrieve-then-read. The stage sits inside the step the whole architecture is named after: it changes what the index sees, then dense, sparse, or hybrid search runs on the rewritten form. Depth for HyDE, multi-query fusion, step-back, and decomposition lives on sibling nodes; what follows owns the family definition, the rewrite-versus-expand line, the technique map, and the structural cost of an extra LLM hop.

How does query rewriting work in a RAG pipeline?

Query rewriting is a pre-retrieval stage: it changes what the search index sees, not how the generator writes. Ma et al. (arXiv:2305.14283, EMNLP 2023) name the frame Rewrite-Retrieve-Read for retrieval-augmented large language models, contrasting it with systems that retrieve on the raw user string and only then read. The rewriter can be a prompted LLM or a smaller trainable model; either way, intent must stay fixed while wording becomes corpus-friendly.

A typical path is four steps. Capture the raw question, which may be vague, pronoun-heavy, or a follow-up that only makes sense with prior turns. Rewrite with an LLM prompt (zero-shot or few-shot) or a small trainable rewriter so one or more search queries preserve intent while adding retrieval-friendly wording. Retrieve by sending each rewritten query to dense search, sparse search, or hybrid BM25-and-vector search. Optionally rerank the shortlist for precision, then generate; rewriting does not replace that post-retrieve stage, which is owned under reranking.

1. Capture the raw question. The user message may be vague, pronoun-heavy, or a follow-up that only makes sense with prior turns.
2. Rewrite. An LLM prompt or a small trainable rewriter emits one or more search queries that preserve intent while adding retrieval-friendly wording (Ma et al., arXiv:2305.14283, EMNLP 2023).
3. Retrieve. Each rewritten query hits dense search, sparse search, or hybrid search. Multiple variants mean multiple retrieval passes unless you fuse them; that fan-out lives on multi-query retrieval and fusion.
4. Optional rerank, then generate. A reranker can reorder the shortlist for precision after retrieval; rewriting does not replace that stage.

Frameworks expose the same stage as query transforms; LlamaIndex’s Query Transform cookbook is the usual API surface as of September 2026. The architectural rule is stable: rewrite first, retrieve on the rewritten form, then generate from the retrieved context. Once that order is fixed, the next distinction is whether you transform the query string or only add terms beside it.

How does query rewriting differ from query expansion?

Query rewriting transforms the query’s structure or wording into a new retrieval string; query expansion keeps the original and adds related terms such as synonyms, morphological variants, and semantically related words (Wikipedia, Query expansion). Both close vocabulary mismatch between how users ask and how documents are written. Classic expansion usually raises recall and can lower precision, because a larger candidate set dilutes the result list unless ranking compensates (Wikipedia’s precision and recall trade-off).

Pseudo-relevance feedback (PRF), treating the top retrieved documents as relevant and expanding from them in the Rocchio tradition, is efficient on average but can damage difficult queries when the first hits are wrong (Wikipedia, citing Amati, Carpineto and Romano on selective application of expansion). LLM-based expansion sidesteps a weak PRF set by generating terms from model knowledge; Jagerman et al. (arXiv:2305.03653, 2023) report that expansions from LLM prompts, especially Chain-of-Thought prompts, can outperform traditional query-expansion methods on MS-MARCO and BEIR. Exact lifts are paper- and corpus-specific; treat any blog percentage without a named benchmark as unverified.

A worked contrast on one underspecified ask, “Best model for customer questions,” makes the difference concrete (Meilisearch’s rewrite-versus-expansion framing, 2026). A rewrite yields a single clearer query such as “best large language models for customer support question-answering.” An expansion keeps the seed and adds related strings such as “LLM for customer support chatbots” and “AI models for customer service automation.” When the failure is identifier-shaped rather than paraphrase-shaped, fix the lexical channel with vocabulary mismatch diagnosis and hybrid search before you blame the rewriter.

What problems does query rewriting solve?

Query rewriting targets failures that start in the query, not in the index. The signatures practitioners hit most often are vocabulary and register mismatch, missing context in multi-turn chat, very short or keyword-only queries, conversational phrasing without retrieval keywords, and compound questions that need split retrieval. Better queries raise the odds that retrieved context grounds the answer; rewriting is not a free hallucination fix if the corpus never contained the fact.

Vocabulary and register mismatch is the classic case: the user says “heart attack” while the corpus says “myocardial infarction,” and both dense and sparse miss until the query speaks document language. Missing antecedents in multi-turn chat produce follow-ups like “Do you have one with SPF?” that carry no topical anchor; rewriting resolves pronouns and prior constraints into a standalone retrieval query before any vector compare (Alhena, multi-turn contextualizer pattern, 2026). Very short product-name queries need expansion into the facets the index actually stores. Conversational phrasing omits the nouns documents use; a rewrite inserts them without changing intent. Multi-part asks need split retrieval, owned in depth by splitting compound questions, and named here because the move still belongs to the rewrite family.

Those five signatures share one property: the index may already hold the gold chunk, yet the raw user string never surfaces it. Once the failure is query-side, the next choice is which transform to apply, because paraphrase, expansion, multi-query, HyDE, step-back, and decomposition are not interchangeable.

What are common query rewriting techniques?

Query rewriting is a family of transforms, not one prompt. The map below is a router: pick the technique that matches the failure signature, then follow the sibling for mechanism depth instead of re-teaching every Advanced RAG trick in one place. Trainable rewriters also exist; Ma et al. (2023) train a small language model as rewriter with reader feedback via reinforcement learning so the rewrite aligns to a frozen black-box LLM, without requiring a full training walkthrough here.

Paraphrase or zero- or few-shot rewrite emits one clearer retrieval query from an LLM prompt and is the default starting point when the ask is almost right but poorly worded. Term expansion adds related terms to the original query and pairs with vocabulary-mismatch diagnosis when synonyms are the gap. Multi-query generation emits several paraphrases, retrieves each, and fuses ranks; that fan-out and fusion live on the multi-query sibling named in the table. Query decomposition splits a compound question into sub-queries. Step-back prompting asks a broader concept question first. HyDE embeds a hypothetical answer and retrieves real neighbours (Gao et al., 2022); both step-back and HyDE rows in the table carry the depth links.

Frameworks such as LangChain and LlamaIndex ship prompt-based variants of the rows above; pick by failure signature, not by blog fashion. Every technique still costs at least one generation call before retrieval, which is the structural price named next without inventing dollar figures.

What does query rewriting cost?

Query rewriting is not free, and the costs are structural rather than a single published price. At least one extra LLM generation runs before every retrieval, so tokens and latency scale with traffic. Vendor reports of roughly 100–300 ms added when rewrite layers run in parallel appear in Alhena’s 2026 FAQ; treat that as one deployment’s figure and measure your own p95 before you rely on it.

Multi-query fan-out multiplies retrieval calls unless variants run concurrently and results are fused; the latency maths belong with multi-query fusion. Semantic drift is the failure where a rewrite changes meaning and searches the wrong neighbourhood (Meilisearch, limitations of query rewriting, 2026). Over-expansion floods the candidate set with too many added terms so precision falls, which is the classic expansion trade-off Wikipedia documents for query expansion.

Those costs decide when rewriting is worth paying, which is a different question from how the stage works. The decision rule is whether the raw user string is already a good search query on a labelled set.

When should you use query rewriting?

Use query rewriting when the raw user string is a bad search query: multi-turn follow-ups that lose antecedents, domain jargon or aliases that still miss after hybrid BM25-and-vector search, long-tail underspecified asks, and compound questions that need split retrieval. The cost order on this site for vocabulary failures is hybrid first, then synonym dictionaries or rewriting, then domain-adapted embeddings under vocabulary mismatch. Rewriting is lever two, not lever zero.

Skip or soft-gate rewriting when the query is already retrieval-shaped and labelled Recall@k is acceptable, when the latency budget cannot absorb an LLM hop, or when your eval set shows the rewriter drifting. Conditional rewriting, only on ambiguous or complex traffic, is the production pattern Meilisearch and similar operator write-ups describe in 2026. Do not adopt their invented lifts; adopt the gating idea and measure on your labels.

Once rewriting is on the critical path, evaluation is the only honest way to keep or drop it. Evaluation names the metrics to compare without re-deriving their formulas.

How do you evaluate query rewriting?

Evaluate rewriting by holding the corpus, embedding model, and top-k fixed and comparing the rewritten query path against the original query path on the same labelled questions. Recall@k and Hit@k ask whether the gold chunk entered the window. MRR and NDCG ask how high the first or graded relevant landed. Answer-level faithfulness checks belong only after retrieval moves; they are generation metrics, not a substitute for retrieval labels.

An optional intermediate signal is reformulation success: the share of rewrite variants that retrieve gold the base query missed (Meilisearch’s evaluation framing, 2026). Do not invent a universal rate for that share. Formulas and worked examples for Recall, MRR and NDCG live on retrieval metrics. Run offline labelled A/B first; move online only after the offline delta is real.

Ma et al. (2023) and Jagerman et al. (2023) show gains on their tasks; your corpus still needs its own numbers. A rewriter that looks good in a demo and fails on labelled aliases is paying latency for semantic drift, which is exactly the failure the cost section warned about.

Query rewriting earns its keep when the user’s words are a poor search query and a measured rewrite lifts Recall@k without drifting intent. Treat expansion as the add-terms subset, keep rewrite-retrieve-read as the pipeline law, and route HyDE, multi-query, step-back, and decomposition to their own nodes when the failure signature demands them. Measure rewrite against baseline on the same labels before every request pays an LLM hop; without that delta, the cheaper path is hybrid search and vocabulary diagnosis, not another prompt.