Query Decomposition for Multi-Part Questions

Query decomposition splits a compound user question into atomic sub-queries, retrieves for each, then synthesizes a final answer from those sub-answers and contexts. It exists for questions that are several facts in disguise: one retrieval step cannot return both Microsoft’s and Google’s profits, or both who directed a film and where that person was born. Haystack’s Advanced RAG: Query Decomposition and Reasoning article and NVIDIA’s RAG Blueprint docs both teach that planner-then-retrieve-then-compose shape. The price is a planner LLM call plus one retrieval per sub-query. The multi-hop loop and stop rules live on multi-hop RAG; sibling paraphrase and abstraction transforms live under multi-query and step-back. Getting the right context to the model still starts from retrieval in RAG.

How does query decomposition work?

Query decomposition is three stages at query time: plan, retrieve, and compose. Haystack states the problem first: a single question can be multiple questions in disguise, and one retrieval step that cannot return both required facts will struggle to produce an accurate final response. The pipeline therefore asks an LLM to emit sub-questions, answers each against the index, then reasons about the final answer from those sub-answers.

Decompose means an LLM planner turns the user question into smaller sub-questions that can be answered in isolation when they are independent. Simple questions stay as one: Haystack’s few-shot keeps “What is the capital of France?” undecomposed. Retrieve per sub-query means each sub-question hits the index on its own, in parallel when independent or chained when the next question needs the previous answer. Synthesize means a second LLM step reasons over the original question plus each sub-answer and context, which is Haystack’s reason-about-the-final-answer step and NVIDIA’s synthesis of intermediate responses.

Worked independent split from Haystack: “Did Microsoft or Google make more money last year?” becomes “How much profit did Microsoft make last year?” and “How much profit did Google make last year?” Worked multi-hop shape from NVIDIA’s Google Frame example returns Naples as the session city only after decomposition supplies the missing hop. Satadru’s Data Engineer Things write-up frames the same failure as semantic dilution: multi-topic queries lose clarity in one chunk set, so each intent must be retrieved independently before synthesis. arXiv:2507.00355 describes the same three-stage pipeline for multi-hop questions whose facts are distributed across documents, with an optional rerank over the merged candidate pool.

Once the three stages are fixed, the design choice that decides latency is whether those sub-queries are independent enough to run in parallel or must chain.

When are sub-queries parallel versus sequential?

The execution mode is the design choice most demos bury inside a single notebook. LangChain YouTube lesson RAG from scratch Part 7 (Query Translation, Decomposition) states the fork plainly: sub-questions can be solved sequentially, or answered independently and then consolidated into a final answer. Parallel fits when sub-queries do not need each other’s answers, such as Microsoft versus Google profit or energy-density comparisons across chemistries. Fire them together to cut wall-clock latency, which is the shape Haystack’s multi-query retrieve component takes for independent splits.

Sequential or chained execution fits when hop two needs an entity from hop one. NVIDIA’s blueprint uses iterative follow-ups with default MAX_RECURSION_DEPTH set to 3, and its multihop first-lady example shows five interconnected steps that cannot be answered as independent lookups. Parallel on a dependent chain still misses the second fact; sequential on independent facts only adds serial latency. Stop criteria and hop budgets for the architecture loop continue on the multi-hop architecture leaf; diagnosis when composition fails lives on multi-hop failures.

Epsilla’s Advanced RAG Optimization note and Nir Diamant’s query-transformations notebook both treat decomposition as one query transform among several; the parallel-versus-sequential choice is what makes decomposition operational rather than a prompt novelty. Wrong mode selection is a common production defect: teams enable a planner, keep a sequential loop for every query, and then blame latency on decomposition itself.

What published gains are reported for query decomposition?

Published numbers are sparse and setting-specific; cite them with units, not as a universal uplift. Petcu et al., arXiv:2510.18633 (Query Decomposition for RAG: Balancing Exploration-Exploitation), also mirrored on the ACL Anthology EACL 2026 long paper page, report that estimating document relevance using rank information and human judgments yields a 35 percent gain in document-level precision, a 15 percent increase in alpha-nDCG, and better performance on long-form generation, under their bandit exploration-exploitation allocation of retrieval budget across sub-queries. That is not a free always-decompose claim; it is a measured gain for how documents are allocated once sub-queries exist.

NVIDIA’s RAG Blueprint docs show a qualitative Google Frame-style multi-hop example: without decomposition the answer can miss the session city; with decomposition the system returns Naples as the reference answer. No percentage is published for that example. arXiv:2507.00355 motivates decomposition for multi-hop questions such as comparing profits across NVIDIA, Apple, and Google in 2023, where relevant facts are distributed across documents, and proposes decompose, retrieve per sub-question, then rerank the merged pool; the abstract does not publish a single leaderboard delta to copy as a product claim.

Do not invent MultiHop-RAG leaderboard numbers that are not in these sources. Prove the pattern on your own labelled multi-hop set; public benchmark reading continues on RAG benchmarks.

What does query decomposition cost?

The costs are structural: extra LLM and retrieval work every time the planner fires. Every complex query pays a planner LLM call before the first retrieve. Each sub-query is its own index search, so retrieval calls scale with the number of sub-questions. Sequential chains add hop latency; parallel independent subs add concurrency load instead. The final synthesis prompt holds many retrieved passages, which raises the risk documented under lost-in-the-middle when too many sub-answers crowd the context window.

NVIDIA’s when-not list calls out time-sensitive lookups such as live stock price because decomposition adds latency by design, and notes that decomposition is unavailable for direct LLM calls when use_kb is false because the feature requires knowledge-base retrieval for subqueries. NVIDIA’s documented default MAX_RECURSION_DEPTH of 3 is a hop budget, not a latency SLA. Exact added p95 depends on planner model, sub-query count, and whether you parallelise.

Those costs decide when the pattern earns its keep and when a simpler rewrite is enough.

When should you use query decomposition?

Use query decomposition when one shot cannot gather every required fact, and skip it when the question is already atomic. NVIDIA’s when-to-use section names multi-hop or complex queries that involve multiple steps or aspects, with intermediate answers combined into a comprehensive final answer. Haystack’s Microsoft versus Google profit example and Data Engineer Things’ semantic-dilution framing are the same multi-aspect class: evidence for each intent lives in different chunks or documents.

NVIDIA’s when-not list skips simple factual questions such as the capital of France, single-concept definitions such as define machine learning, highly specific syntax questions, and real-time single-value queries where extra hops only hurt latency. Haystack’s few-shot that leaves capital-of-France undecomposed is the same skip rule from the planner side. When the system still fails after decomposition, diagnose on composition failure diagnosis; when you need the full retrieve-reason-retrieve loop with stop rules, use multi-hop stop rules.

Prefer multi-query retrieval when the need is one fact under many phrasings. Prefer step-back prompting when the need is background principles for an over-constrained ask rather than multiple atomic facts.

How is query decomposition different from multi-query and step-back?

Three query-time transforms get conflated in practitioner menus. They solve different problems. Query decomposition splits one question into different atomic information needs, then composes the answers. Multi-query or fusion generates paraphrase variants of the same need and merges ranked lists, often with reciprocal rank fusion, with depth on multi-query and fusion. Step-back asks a more general question first to retrieve background principles, then answers the specific ask, with depth on step-back prompting for retrieval.

The broader rewrite and expansion family sits on query rewriting. GitHub notebooks and Epsilla guides that stack decomposition with other query transforms are describing a menu, not a default to enable every request. Medium LangChain decomposition walkthroughs sometimes reuse MultiQueryRetriever naming; that class is paraphrase-oriented, so true atomic splits need a dedicated sub-question planner rather than a paraphraser alone.

Pick the failure you are fixing: multi-fact composition, paraphrase mismatch, or over-constrained phrasing, then measure whether the composed answer actually moves on labelled multi-hop questions.

How do you implement query decomposition?

The pattern is planner prompt, retrieve per sub-query, then synthesize, with a hop cap on sequential paths. Haystack’s cookbook is the parallel-then-reason shape with a structured Questions schema and a multi-query retriever over sub-questions. NVIDIA’s blueprint toggles ENABLE_QUERY_DECOMPOSITION with MAX_RECURSION_DEPTH. LangChain demos that only paraphrase should not be mistaken for atomic decomposition; prefer a dedicated sub-question planner such as LlamaIndex SubQuestion-style engines when the sub-queries are different information needs. Nir Diamant’s query-transformations notebook shows a decompose-into-2-to-4-sub-queries prompt shape that is useful as a starting planner template.

Per-sub retrieval works against stores such as Weaviate, Qdrant, Pinecone, or Milvus. Weaviate is the default store example on this site when each sub-query may need hybrid keyword and vector search before synthesis, using patterns documented in Weaviate’s search guides as of September 2026; that does not make Weaviate required for decomposition. Cap hops the way NVIDIA documents with MAX_RECURSION_DEPTH of 3 unless your labelled set proves a deeper chain is necessary.

The runnable, pinned end-to-end pipeline belongs on building the pipeline; hop termination belongs on multi-hop loop termination; the retrieval cluster map is getting the right context to the model. What matters after the planner is wired is logging sub-query count, parallel versus sequential choice, and whether precision on multi-aspect questions moves without blowing p95 on simple traffic.

Query decomposition is planner-driven splitting of compound questions into atomic sub-queries, per-sub retrieval, and synthesis, as Haystack and NVIDIA document and as Petcu et al. measure under bandit document allocation. Use parallel execution for independent facts and sequential hops when entities must chain, cite the 35 percent precision and 15 percent alpha-nDCG gains only in that bandit setting, and skip the planner on already-atomic lookups. Route paraphrase fusion and abstraction rewrites to their sibling pages, and keep hop-loop depth on the multi-hop architecture leaf.