Query Routing: Sending Each Query to the Right Source

Query routing analyses the user question and selects which predefined path to take: which data source, index, retrieval strategy, tool, or no-retrieval path, instead of always running one retrieve-then-generate stack. Rogiia’s Build an Advanced RAG App: Query Routing guide on DEV Community defines it as taking the query and deciding the next action from a list of predefined choices. Sami Maameri’s Towards Data Science article Routing in RAG-Driven Applications and Jason Liu’s write-up of Anton Troynikov’s Chroma session both treat routing as the control plane once you have more than one place to look. The price is a cheap classifier, embedding lookup, or small LLM call, plus the damage of wrong routes if you ship without fallbacks. Related policies live on adaptive retrieval, self-query, and agentic patterns. Getting the right context to the model still starts from retrieval in RAG.

Why does one-size-fits-all retrieval fail?

Always searching every store, or always using the same strategy, fails in structural ways that show up across the live ranking set. Rogiia opens with the fork: for simple RAG you retrieve from a single data source, but complex apps have multiple data sources or need different types of context for different queries, so a one-size-fits-all stack either misses the right store or pays for every store on every question. Guild.ai’s Query Routing glossary names the same pressure as cost control at scale, accuracy and relevance, and latency when the wrong path runs.

Anton Troynikov, as reported by Jason Liu in September 2025, attacks the “big pile of records” default: one giant index for all application data. That default creates security and compliance issues when user data is only separated by filters, reduced recall because filtering approximate nearest-neighbour graphs wastes compute on irrelevant nodes, and write-performance cost on an already giant index. Filtering reduces recall in vector search because approximate graphs trade recall for speed, and pre-filtering or post-filtering both spend budget poorly when selectivity is high. Separate indexes plus routing exist to avoid that pile, not to add ceremony.

Milvus Blog’s Beyond Naive RAG piece frames a four-node modern architecture where query routing asks whether the query needs retrieval at all before rewriting, strategy selection, and generation. AWS Builder’s Part 6 on query routing, multi-hop, and agentic RAG places routing as Pattern 1 when naive always-retrieve breaks down. Wrong modality is the third failure: order IDs and exact tokens need keyword or structured lookup, while vague product asks need dense search, and some questions should not retrieve at all, which continues on the adaptive skip gate.

What can a query router choose among?

You pre-define the menu. Rogiia stresses that each choice needs a clear description, because that text is what the selector reads when it decides. Vague descriptions produce misroutes even when the underlying indexes are good. The live guides enumerate four classes of choice that cover most production menus.

Different data sources are the first class: product documentation versus return policies, or a research paper versus a recipe corpus in Rogiia’s worked LlamaIndex example that also offers a Google search tool. Different indexes or strategies on the same corpus are the second class: keyword versus vector, summary versus precise vector, or sentence-window versus parent-document styles, so the router picks how to search rather than only where. Strategy depth continues on sentence-window retrieval and parent-document retrieval.

Tools and APIs are the third class: web search or other HTTP tools when the answer is not in the corpus, which Rogiia demonstrates with a Google search path beside document engines. No retrieval or direct LLM is the fourth class: a fixed menu entry that overlaps the adaptive skip gate, kept here only when “answer without retrieval” is an explicit route rather than a learned skip policy. LlamaIndex’s RouterQueryEngine routes among query engines wrapped as QueryEngineTool objects with those descriptions, which is the shape Rogiia’s single-source LLM selector demo implements.

Once the menu exists, the next design question is which mechanism picks a choice, because router types differ in latency shape more than in vocabulary.

What types of query routers exist?

Router types differ in how the decision is made, not in what a choice is. Rogiia and Maameri both catalogue the same families. An LLM selector or completion router puts choice descriptions and the user query into a prompt and lets a completion model name the route, which is Rogiia’s LLM Selector Router and Maameri’s LLM Completion Router that returns a single word used by an if-else branch. An LLM function-calling or Pydantic router phrases each choice as a tool or structured schema so a tool-calling model returns the selected function, which is how LlamaIndex Pydantic routers and most agent tool picks work, and how Rogiia’s demo wires QueryEngineTool objects into a Router Query Engine.

A semantic router embeds the user query and matches it to example utterances per route, then picks the nearest example path. Maameri notes that this type should be faster than LLM-based routers because it requires a single index query rather than a generation call. Rogiia describes the same exemplar-embedding nearest-neighbour rule. A keyword or rule router matches simple patterns such as the word return to a policy store; Rogiia and Maameri both treat it as the cheap, brittle option that needs no model. A zero-shot or fine-tuned small classifier uses a small model whose only job is route labels; Rogiia lists that family separately from full LLM selectors and notes small models are more cost-effective for classification. Language classification routers appear when corpora are language-split.

Milvus Blog’s Node 3, retrieval strategy selection, is the same decision space when the menu is hybrid versus dense versus keyword rather than separate document stores. Guild.ai groups the mechanisms as logical LLM-based routing, semantic embedding-based routing, and rule-based or hybrid routing. Single versus multi selection is orthogonal to the family above: the same LLM selector can be a single selector or a multi selector.

Those type choices decide whether you pay an embedding lookup or a generation call before retrieval starts, which leads to the multiplex-versus-select decision.

Should you route to one path or search everything?

Three regimes appear on the live ranking set; pick explicitly rather than accidentally doing all three. Full multiplexing, as Anton describes in Liu’s write-up, sends the query to all applicable data sources and combines results. Longer context windows and lower per-token costs make sending more candidates to the model cheaper than before, and Anton argues re-ranking models are underrated when used as an augmentation to query routing rather than as a fix for one giant index. Multiplex maximises recall and spend.

Single-choice routing picks one path. Rogiia’s single selector and LlamaIndex single selectors minimise spend and can miss answers that span two sources. Multi-choice routing picks several paths and combines them when a question spans topics, which Rogiia contrasts with single choice in a dedicated section: retrieve from many data sources and consolidate into one final answer. LLM routing in Anton’s framing is the classification task of deciding which sources are relevant; he notes you do not need the latest model for that classification and that you must calibrate the LLM judge against what a human would choose on real traffic.

Microsoft Q&A threads on dynamic query routing for multi-source answer generation ask the same operational question in enterprise settings: how to decide which knowledge bases participate without always paying the full fan-out. Fusion mechanics for paraphrase variants of one need, a different job from source selection, continue on multi-query retrieval.

What does query routing cost?

The costs are the router overhead and the routes you still take; the savings are the full-RAG paths and irrelevant stores you skip. Rule routers add near-zero model cost. Semantic routers pay one embedding lookup, which Maameri contrasts with LLM routers that require a generation call. LLM completion and function-calling routers add that generation latency on every query before retrieval starts. Guild.ai lists cost control, accuracy, and latency as the three reasons routing matters at scale, without publishing a universal millisecond table.

Multiplexing spends retrieval and context tokens across every selected source; single-choice routing spends on one path and risks empty context on a misroute. Misroute cost is empty or noisy context that still looks fluent after generation. Fallback chains escalate to a broader search or a human path when confidence is low; Rogiia’s selector_result attribute exists so you can inspect which tool was chosen and why. Exact milliseconds and dollars are not published as an industry constant on this ranking set; measure router-on versus always-multiplex on your labelled query-to-route pairs under RAG evaluation.

Adjacent research on routing among retrieval-augmented LLMs, rather than among indexes, is a different cost story. arXiv:2505.23052 and the NeurIPS 2025 RAGRouter poster report routing across models under RAG with gains of 1.67 percent to 9.33 percent over the best individual LLM and non-RAG-aware routers; that is model-pool routing after documents exist, not source selection before retrieval.

When should you use query routing?

Use query routing when you have two or more meaningfully different sources, indexes, or strategies and traffic is mixed. Rogiia’s product-versus-policy and document-versus-web-search examples are the strong-fit shape. Anton’s denormalized one-index-per-user-per-source design requires routing as soon as queries must find the right small index. Milvus Blog’s four-node architecture expects a routing node before rewrite and hybrid retrieval. AWS Builder’s Pattern 1 likewise places routing first when naive RAG breaks down on heterogeneous traffic.

Skip routing when one homogeneous corpus and one strategy is enough; a router only adds failure modes. Pitfalls that show up in the guides are brittle keyword lists, vague route descriptions, no fallback on low confidence, and conflating route-with-filter on one index with route-to-index across stores. Self-query depth for metadata filters on one index continues on self-query retrieval. Full plan, tool, and reflect loops continue on agent tool loops.

Build order that matches the live tutorials is heuristics or keyword rules first while you collect labelled examples, then a semantic or small-classifier router once the menu is stable, then an LLM selector only when description-based reasoning beats exemplar matching on your traffic.

How is query routing different from adaptive retrieval or self-query?

These sit near each other and answer different questions. Query routing chooses which predefined path among sources, indexes, tools, or strategies. Adaptive retrieval decides whether to retrieve at all, with depth on whether-to-retrieve decisions. Self-query writes a metadata filter plus a semantic query against one index, with depth on metadata self-query filters. Adaptive-RAG chooses no, single, or multi-step retrieval by query complexity, with depth on Adaptive-RAG. Agentic RAG may own tool selection as an ongoing plan and reflect loop, with depth on open agent tool selection.

Maameri notes that agents already perform routing when they select tools via function calling, which is why LLM function-calling routers and agent tool picks look alike. The operational border for this leaf is fixed-menu path selection before retrieval, not an open-ended agent loop. RAGRouter-style papers that route among LLMs after retrieval are a third neighbour: they optimise which generator reads the documents, not which index supplied them.

Keep the borders sharp in production menus. A single “smart router” that also rewrites, also skips, and also plans multi-hop steps becomes undebuggable; separate the decisions the way Milvus Blog separates its four nodes.

How do you implement query routing?

Define routes with clear descriptions, pick a router type, wire each choice to a query engine or retriever, log route and confidence, and add a fallback. Rogiia’s LlamaIndex demo builds QueryEngineTool objects for each document engine and a Google search tool, attaches descriptions, then creates a Router Query Engine with an LLM selector in single-selector mode so only one tool fires. Maameri shows completion routers that emit a route word into an if-else, function-calling routers that return a selected function, and semantic routers that nearest-neighbour match exemplar embeddings.

Per-route indexes work with stores such as Weaviate, Qdrant, Pinecone, and Chroma. Weaviate is the default store example on this site when each route may need hybrid keyword and vector search inside its own collection, using patterns documented in Weaviate’s search guides as of September 2026; that does not make Weaviate required for routing, and separate collections or tenants are a common way to avoid the big-pile filter tax Anton describes. Hybrid retrieval inside a selected route continues on hybrid search.

The runnable, pinned end-to-end pipeline belongs on building the pipeline; measure under evaluation; the retrieval cluster map is getting the right context to the model. What matters after the selector is wired is logging selector_result-style choices, calibrating LLM judges on human labels as Anton urges, and proving that multiplex-off saves spend without collapsing recall on multi-source questions.

Query routing is fixed-menu path selection for heterogeneous RAG: choose among sources, indexes, tools, or no-retrieval with an LLM, semantic, keyword, or classifier router, then retrieve only on the selected paths unless you deliberately multiplex. Use it when one pile of records or one always-on strategy wastes work or hurts recall; skip it on a single homogeneous corpus; and keep adaptive skip, self-query filters, Adaptive-RAG depth, and agent loops on their sibling leaves rather than collapsing every control decision into one opaque router.