Query Routing: Sending Each Query to the Right Source

Query routing analyses the user question and selects which predefined path to take — which data source, index, retrieval strategy, tool, or no-retrieval path — instead of always running one retrieve-then-generate stack (ruxu.dev; LlamaIndex Router Modules). It usually sits after rewrite or guardrails and before retrieval. The price is a cheap classifier or small LLM call, plus the damage of wrong routes if you ship without fallbacks. Related policies live on adaptive retrieval, self-query, and agentic RAG; this page is the fixed-choice router.

Why does one-size-fits-all retrieval fail?

Always searching every store — or always using the same strategy — fails in three structural ways.

• Noise. Wrong-source chunks distract generation. Answering a refund-policy question does not need product-spec chunks in the same prompt (Wicked Smart Data).
• Cost and latency. If most questions belong to one topic but every query hits every source, you multiply work — Wicked Smart Data’s framing is on the order of 5× retrieval when ~60% of traffic is one topic still fan-out to every store.
• Wrong modality. Order IDs need SQL or keyword lookup; vague product asks need dense search (Wicked Smart Data; Dhanu strategy list). Some questions should not retrieve at all — that skip gate continues on adaptive retrieval.

What can a query router choose among?

You pre-define the menu. Each choice needs a clear description — that text is what the selector reads (ruxu.dev).

• Different data sources. Policies versus products versus inventory APIs (Wicked Smart Data e-commerce triage).
• Different indexes or strategies on the same corpus. Keyword versus vector; summary versus precise vector; sentence-window versus parent-document styles (ruxu; LlamaIndex summary-vs-vector Router Query Engine example). Strategy depth continues on sentence-window and parent-document.
• Tools and APIs. Web search, weather, or other HTTP tools when the answer is not in the corpus (ruxu Google Search tool).
• No retrieval / direct LLM. Overlaps the adaptive skip gate; route it here only as a fixed choice on the menu.

LlamaIndex’s RouterQueryEngine routes among query engines wrapped as QueryEngineTools with those descriptions.

What types of query routers exist?

Router types differ in how the decision is made — not in what a “choice” is. Five families show up across live guides:

• LLM text / completion selectors. Dump choice descriptions into a prompt and let a completion model name the route (LlamaIndex LLMSingleSelector; ruxu LLM Selector Router).
• LLM function-calling / Pydantic selectors. Pass choices as structured schemas to a tool-calling endpoint (LlamaIndex PydanticSingleSelector / PydanticMultiSelector; Towards Data Science DataSource Pydantic object).
• Semantic routers. Embed the query and match it to example utterances per route (ruxu; TDS) — typically one embedding lookup rather than a generation call.
• Keyword / rule routers. Pattern lists such as “return” → policy store (TDS HR/Finance keywords; Wicked Smart Data simple_rule_router) — fast and brittle on paraphrase.
• Small classifiers. A finetuned DistilBERT-class model or zero-shot classifier whose only job is route labels (ruxu; Dhanu AskSolique DistilBERT path under 15 ms observation).

Language routers appear when corpora are language-split (ruxu). Single versus multi selection is orthogonal to the family above.

Should you route to one path or search everything?

Three regimes appear in live pages — pick explicitly; do not accidentally do all three.

• Full multiplexing. Send the query to all applicable sources and fuse or re-rank (Jason Liu / Anton, Chroma, September 2025). Maximises recall and spend; longer contexts make this cheaper than it used to be.
• Single-choice routing. Pick one path (ruxu single selector; LlamaIndex single selectors). Minimises spend; can miss answers that span two sources.
• Multi-choice routing. Pick several paths and combine (ruxu multiple choice; LlamaIndex multi selectors).

Calibrate LLM judges on real traffic labels (jxnl / Anton). Fusion mechanics for paraphrase variants — a different job — continue on multi-query retrieval.

What does query routing cost?

The costs are the router overhead and the routes you still take — the savings are the full-RAG paths you skip.

• Router overhead. Rule routers add near-zero model cost. Semantic or classifier routers can sit in the low tens of milliseconds — Dhanu (AskSolique, 10 February 2026) reports DistilBERT under 15 ms. LLM routers typically add on the order of 200–500 ms with a fast model (Wicked Smart Data tip — measure yours).
• Savings from not searching everything. Skipping irrelevant stores cuts the multiplex tax (Wicked Smart Data). AskSolique’s observed mix was roughly 40% no-RAG, 25% simple factual, 35% full pipeline — one product’s telemetry, not an industry average.
• Misroute cost. Wrong source → empty or noisy context. Fallback chains escalate; Dhanu reports about 12% of queries escalating at least once on that system.

When should you use query routing?

Use query routing when you have two or more meaningfully different sources, indexes, or strategies and traffic is mixed — and skip it when one homogeneous corpus and one strategy is enough.

• Strong fit: multiple sources; summary versus precise vector on the same documents (LlamaIndex); exact-ID lookups mixed with semantic asks (ruxu); cost or latency pressure from treating every query as full RAG (Dhanu).
• Weak fit / pitfalls: a single corpus with one retriever (a router only adds failure modes); brittle keyword lists; vague route descriptions; no fallback on low confidence; conflating route-with-filter (self-query) with route-to-index.

Dhanu’s build order is heuristics first, then a classifier once you have hundreds of labelled examples. Full plan/tool/reflect loops continue on agentic RAG.

How is query routing different from adaptive retrieval or self-query?

These sit near each other and answer different questions.

• Query routing chooses which predefined path (source, index, tool, or strategy) — this page.
• Adaptive retrieval decides whether to retrieve at all → adaptive retrieval.
• Self-query writes a metadata filter plus a semantic query against one index → self-query retrieval.
• Adaptive-RAG chooses no / single / multi-step retrieval by query complexity → Adaptive-RAG.
• Agentic RAG may own tool selection as an ongoing plan/reflect loop → agentic RAG.

How do you implement query routing?

Define routes with clear descriptions; pick a router type; wire each choice to a query engine or retriever; log route and confidence; add a fallback. LlamaIndex’s canonical path is RouterQueryEngine with QueryEngineTool descriptions and a PydanticSingleSelector, LLMSingleSelector, or multi-selector variant (live docs, as of July 2026). Keyword, LLM, and semantic chains appear in TDS and Wicked Smart Data tutorials. Per-route indexes work with stores such as Weaviate, Qdrant, Pinecone, and Chroma. For a minimal runnable pipeline see building the pipeline; measure under evaluation; for the retrieval cluster see retrieval.