Query Decomposition for Multi-Part Questions

Query decomposition splits a compound user question into atomic sub-queries, retrieves for each, then synthesizes a final answer from those sub-answers and contexts. It exists for questions that are several facts in disguise — one retrieval step cannot return both Microsoft’s and Google’s profits, or both “who directed Inception” and “where was that person born.” The price is a planner LLM call plus one retrieval per sub-query. The multi-hop loop and stop rules live on multi-hop RAG; this page is the split itself.

How does query decomposition work?

Query decomposition is three stages at query time — plan, retrieve, compose.

1. Decompose. An LLM planner turns the user question into smaller sub-questions that can be answered in isolation when they are independent. Simple questions stay as one: Haystack’s few-shot keeps “What is the capital of France?” undecomposed (blog, 30 September 2024).
2. Retrieve per sub-query. Each sub-question hits the index on its own — in parallel when independent, or chained when the next question needs the previous answer.
3. Synthesize. A second LLM step reasons over the original question plus each sub-answer/context and returns the composed answer (Haystack “reason about the final answer”; NVIDIA RAG Blueprint synthesis step).

Worked independent split (Haystack): “Did Microsoft or Google make more money last year?” becomes “How much profit did Microsoft make last year?” and “How much profit did Google make last year?” Worked dependent chain (Qdrant docs): “Where was the director of the film Inception born?” retrieves film context, then a follow-up “Where was Christopher Nolan born?” — evidence that never co-occurred in one chunk.

When are sub-queries parallel versus sequential?

The execution mode is the design choice most guides bury inside a single demo.

• Parallel (independent). Sub-queries do not need each other’s answers — energy density of solid-state vs lithium-ion, or Microsoft vs Google profit. Fire them together to cut wall-clock latency (smartfaqs execution strategies; Haystack multi-question retrieve).
• Sequential / chained (dependent). Hop two needs an entity from hop one. Qdrant’s guide builds on Self-Ask (arXiv:2210.03350) and IRCoT (arXiv:2212.10509): retrieve, ask what is still missing, or stop with DONE, capped at MAX_HOPS = 3. NVIDIA’s blueprint uses iterative follow-ups with default MAX_RECURSION_DEPTH=3.

Parallel on a dependent chain still misses the second fact; sequential on independent facts only adds serial latency. Stop criteria and hop budgets for the architecture loop continue on multi-hop RAG; diagnosis when composition fails lives on multi-hop failures.

What published gains are reported for query decomposition?

Published numbers are sparse and setting-specific — cite them with units, not as a universal uplift.

• Petcu et al., arXiv:2510.18633 (Query Decomposition for RAG: Balancing Exploration-Exploitation): under a fixed retrieval budget, estimating document relevance with rank information and judgments yields a 35% gain in document-level precision and a 15% increase in α-nDCG versus naïve allocation, with better downstream long-form generation on their NeuCLIR / ResearchyQuestions setup. Hierarchical expansion of informative sub-queries yields a 30% precision gain over selecting all documents from a single-level decomposition (paper results sections).
• NVIDIA RAG Blueprint (docs). On a Google Frame-style multi-hop question about Pompeii / UNESCO, the system without decomposition answers with a year but misses the session city; with decomposition it returns “Naples.” That is a qualitative example — no percentage is published.

Do not subtract these figures from other multi-hop leaderboards. Prove the pattern on your own labelled multi-hop set; public benchmark reading continues on RAG benchmarks.

What does query decomposition cost?

The costs are structural — extra LLM and retrieval work every time the planner fires.

• Planner call. Every complex query pays an LLM call before the first retrieve.
• N retrievals. Each sub-query is its own index search, so calls scale with the number of sub-questions.
• Serial vs fan-out latency. Sequential chains add hop latency; parallel independent subs add concurrency load instead.
• Heavier synthesis context. The final prompt holds many retrieved passages — production guides commonly cap decomposition around 3–5 sub-queries to limit lost-in-the-middle risk (smartfaqs FAQ).

NVIDIA’s when-not list calls out time-sensitive lookups (for example live stock price) because decomposition adds latency by design. Qdrant’s sample loop caps hops at 3.

When should you use query decomposition?

Use query decomposition when one shot cannot gather every required fact — and skip it when the question is already atomic.

• Strong fit: multi-hop chains and multi-aspect comparisons where evidence lives in different chunks or documents (Haystack; NVIDIA; Qdrant; smartfaqs).
• Skip: simple factual lookups (“capital of France”), single-concept definitions, highly specific syntax questions, and real-time single-value queries where extra hops only hurt latency (NVIDIA when-not list). FAQ-style corpora often want a router that fires decomposition only when the planner detects a multi-part need (smartfaqs).

When the system still fails after decomposition, diagnose on multi-hop failures; when you need the full retrieve→reason→retrieve loop with stop rules, use multi-hop RAG.

How is query decomposition different from multi-query and step-back?

Three query-time transforms get conflated on SERPs. They solve different problems.

• Query decomposition splits one question into different atomic information needs, then composes the answers.
• Multi-query / fusion generates paraphrase variants of the same need and merges ranked lists (often with RRF). Depth on multi-query retrieval.
• Step-back asks a more general question first to retrieve background principles, then answers the specific ask. Depth on step-back prompting.

The broader rewrite/expansion family sits on query rewriting. Guides that stack all three (dev.to query transformation overview) are describing a menu — not a default to enable every request.

How do you implement query decomposition?

The pattern is planner prompt → retrieve per sub-query → synthesize, with a hop cap on sequential paths. Haystack’s cookbook is the parallel-then-reason shape; Qdrant’s docs show the sequential Self-Ask loop with RRF across hops; NVIDIA’s blueprint toggles ENABLE_QUERY_DECOMPOSITION with MAX_RECURSION_DEPTH. LangChain’s MultiQueryRetriever is paraphrase-oriented — for true atomic splits prefer a dedicated sub-question planner (LlamaIndex’s SubQuestion-style engines are the usual reference). Per-sub retrieval works against stores such as Weaviate, Qdrant, Pinecone, or Milvus. For a minimal runnable pipeline see building the pipeline; for hop termination see multi-hop RAG; for the retrieval cluster see retrieval.