Query Rewriting and Expansion for RAG

Query rewriting reformulates the user’s question into one or more retrieval-friendly queries before search — clarifying, expanding, paraphrasing, or splitting it — so the retriever matches how answers are phrased in the corpus. Expansion is the add-terms subset of that family; rewriting is the broader transform. The pipeline shape is rewrite-retrieve-read (Ma et al., EMNLP 2023), not naive retrieve-then-read. This page covers how the stage works, how it differs from classic expansion, which technique to reach for, and what it costs — with depth for HyDE and multi-query on their own nodes under retrieval.

How does query rewriting work in a RAG pipeline?

Query rewriting is a pre-retrieval stage: it changes what the search index sees, not how the generator writes. A typical path is four steps:

1. Capture the raw question. The user message may be vague, pronoun-heavy, or a follow-up that only makes sense with prior turns.
2. Rewrite. An LLM prompt (zero-shot or few-shot) or a small trainable rewriter emits one or more search queries that preserve intent while adding retrieval-friendly wording — the Rewrite-Retrieve-Read frame Ma et al. (arXiv:2305.14283, EMNLP 2023) introduced for retrieval-augmented LLMs.
3. Retrieve. Each rewritten query hits dense search, sparse search, or hybrid search. Multiple variants mean multiple retrieval passes unless you fuse them — that fan-out lives on multi-query retrieval.
4. Optional rerank, then generate. A reranker can reorder the shortlist for precision after retrieval; rewriting does not replace that stage (reranking).

Frameworks expose the same stage as query transforms (LlamaIndex’s Query Transform cookbook is the usual API surface). The architectural rule is stable as of July 2026: rewrite first, retrieve on the rewritten form, then generate from the retrieved context.

How does query rewriting differ from query expansion?

Query rewriting transforms the query’s structure or wording into a new retrieval string; query expansion keeps the original and adds related terms — synonyms, morphological variants, and semantically related words (Wikipedia, Query expansion).

Both close vocabulary mismatch. Classic expansion usually raises recall and can lower precision, because a larger candidate set dilutes the result list unless ranking compensates (Wikipedia’s precision/recall trade-off). Pseudo-relevance feedback (PRF) — treating the top retrieved documents as relevant and expanding from them, in the Rocchio tradition — is efficient on average but can damage difficult queries when the first hits are wrong (Wikipedia, citing Amati, Carpineto and Romano on selective application of expansion). LLM-based expansion sidesteps a weak PRF set by generating terms from model knowledge; Jagerman et al. (arXiv:2305.03653, 2023) report that expansions from LLM prompts — especially Chain-of-Thought prompts — can outperform traditional query-expansion methods on MS-MARCO and BEIR. Exact lifts are paper- and corpus-specific; treat any blog percentage without a named benchmark as unverified.

A worked contrast on one underspecified ask — “Best model for customer questions” — makes the difference concrete (Meilisearch’s rewrite-vs-expansion framing, 2026):

• Rewrite → a single clearer query such as “best large language models for customer support question-answering”.
• Expand → keep the seed and add related strings (“LLM for customer support chatbots”, “AI models for customer service automation”).

When the failure is identifier-shaped rather than paraphrase-shaped, fix the lexical channel with vocabulary mismatch diagnosis and hybrid search before you blame the rewriter.

What problems does query rewriting solve?

Query rewriting targets failures that start in the query, not in the index. The signatures practitioners hit most often:

• Vocabulary and register mismatch. The user says “heart attack”; the corpus says “myocardial infarction”. Dense and sparse both miss until the query speaks document language (vocabulary mismatch).
• Missing context in multi-turn chat. Follow-ups like “Do you have one with SPF?” carry no topical anchor. Rewriting resolves pronouns and prior constraints into a standalone retrieval query before any vector compare (Alhena, multi-turn contextualizer pattern, 2026).
• Very short or keyword-only queries. A bare product name needs expansion into the facets the index actually stores.
• Conversational phrasing without retrieval keywords. Informal questions omit the nouns documents use; a rewrite inserts them without changing intent.
• Compound questions. Multi-part asks need split retrieval — owned in depth by query decomposition, named here because it is still a rewrite-family move.

Better queries raise the odds that retrieved context grounds the answer. Rewriting is not a free hallucination fix if the corpus never contained the fact — it only removes query-side miss as the cause.

What are common query rewriting techniques?

Query rewriting is a family of transforms, not one prompt. Use this map to pick a technique, then follow the sibling page for mechanism depth — this page does not re-teach each Advanced RAG trick.

Trainable rewriters also exist: Ma et al. (2023) train a small language model as rewriter with reader feedback via reinforcement learning so the rewrite aligns to a frozen black-box LLM. Frameworks (LangChain, LlamaIndex) ship prompt-based variants of the rows above; pick by failure signature, not by blog fashion.

What does query rewriting cost?

Query rewriting is not free, and the costs are structural:

• At least one extra LLM generation before every retrieval. You pay tokens and latency on every rewritten request. Vendor reports of roughly 100–300 ms added when rewrite layers run in parallel appear in Alhena’s 2026 FAQ — treat that as one deployment’s figure and measure your own p95 before you rely on it.
• Multi-query fan-out multiplies retrieval calls unless variants run concurrently and results are fused — the latency maths belong on multi-query retrieval.
• Semantic drift. A rewrite that changes meaning searches the wrong neighbourhood (Meilisearch, limitations of query rewriting, 2026).
• Over-expansion. Too many added terms flood the candidate set and precision falls — the classic expansion trade-off Wikipedia documents for query expansion.

When should you use query rewriting?

Use query rewriting when the raw user string is a bad search query: multi-turn follow-ups that lose antecedents, domain jargon or aliases that still miss after hybrid search, long-tail underspecified asks, and compound questions that need split retrieval. Skip or soft-gate it when the query is already retrieval-shaped and labelled Recall@k is acceptable, when the latency budget cannot absorb an LLM hop, or when your eval set shows the rewriter drifting. The cost order on this site for vocabulary failures is hybrid first, then synonym dictionaries or rewriting, then domain-adapted embeddings (vocabulary mismatch) — rewriting is lever two, not lever zero.

How do you evaluate query rewriting?

Evaluate rewriting by holding the corpus, embedding model, and top-k fixed and comparing the rewritten query path against the original query path on the same labelled questions.

• Recall@k / Hit@k — did the gold chunk enter the window?
• MRR / NDCG — how high did the first (or graded) relevant land?
• Answer-level checks — only after retrieval moves; faithfulness of the written answer is a generation metric, not a substitute for retrieval labels.
• Reformulation success — optional intermediate: share of rewrite variants that retrieve gold the base query missed (Meilisearch’s evaluation framing, 2026). Do not invent a universal rate.

Formulas and worked examples for Recall, MRR and NDCG live on retrieval metrics. Offline labelled A/B first; online only after the offline delta is real. Ma et al. (2023) and Jagerman et al. (2023) show gains on their tasks — your corpus still needs its own numbers.