Retrieval in RAG: Getting the Right Context to the Model

Retrieval in RAG scores indexed chunks against the query and returns the passages the model will see. If the right passage is missing from that set, the answer cannot be right, no matter how strong the generator is. AWS’s “What is RAG” documentation and LangChain’s retrieval docs both frame the same beat: fetch relevant external knowledge at query time, then put it in the prompt. Weaviate’s introduction to RAG (as of September 2026) separates that online retrieve-and-augment path from the offline index that made search possible. Scoring can be dense similarity, sparse lexical match, or both fused; optional query transforms rewrite what is searched before any score runs; top-k decides how many chunks enter the prompt. This hub maps those decisions and routes every technique under retrieval to the leaf that owns the depth.

How does retrieval work in RAG?

Retrieval in RAG takes the user query, scores indexed chunks by dense similarity, sparse lexical match, or both, ranks them, and returns a top-k set that becomes the model’s context. IBM’s RAG overview and Databricks’ end-to-end RAG workflow article walk the same retrieve-then-augment sequence after the corpus has already been chunked, embedded and indexed. Wikipedia’s retrieval-augmented generation entry (as of 2026) still centres Lewis et al. 2020 (arXiv:2005.11401) as the paper that named the method: non-parametric memory retrieved at inference time, not only parametric weights from training. AWS’s documentation uses the same retrieve-then-augment language for the online path, and LangChain’s retrieval docs describe fetching external knowledge at query time as the retriever’s job inside that path.

The stages, in order, are narrow. An optional query transform rewrites, expands, invents a hypothetical answer, fans out variants, decomposes a multi-part question, or lets the model emit a structured filter before anything is scored. Score runs dense approximate nearest-neighbour search over embeddings, and/or lexical BM25 over terms: dense finds paraphrases; the sparse channel finds exact tokens. When both channels run, the ranked lists fuse into one ranking, usually reciprocal rank fusion or a normalised weighted blend. Select top-k keeps the shortlist the model will actually see. Hand-off puts those chunks into the prompt for generation. A second-stage cross-encoder that reorders that shortlist is reranking, not retrieval itself. Meilisearch’s catalogue of RAG types and Microsoft’s techniques list both still name several of those stages as separate “types”; under a stage map they are options inside retrieve, not competing architectures.

What gets scored and what gets returned are the only decisions that matter at this stage. Everything upstream (chunking, embeddings, indexing) sets the ceiling on what can be found. Everything downstream (rerank, generate) can only work with what retrieval handed over. Upstream borders live on the what it does not fix root map and the pipeline hub; the retrieval job itself is the score, the fuse and the cut-off.

The figure is the map at a glance. The practical split that follows is why answer quality tracks retrieval quality so tightly, and which failure modes appear when the wrong channel or the wrong cut-off is used.

Why does retrieval quality determine the answer?

Retrieval quality determines the answer because the model only sees the chunks that came back. A faithful answer to the wrong context is still wrong. Databricks’ RAG FAQ (as of 2026) names poor retrieval quality as the most common RAG failure mode, and states that a strong generator cannot compensate for irrelevant chunks. Microsoft’s February 2025 common RAG techniques post makes the same operational point: techniques that improve what is retrieved move answer quality more than prompt tweaks alone once the index exists. K2view’s practical guide puts the same claim in product language: the gap between a demo and production lives almost entirely in retrieval quality, because a model can only answer well from context it actually receives.

Dense-only search fails where exact tokens matter. Product codes, error numbers, statute IDs and rare names are the cases hybrid guides repeatedly list as the dense blind spot: BM25 catches the literal string; the embedding often does not. Embeddings find similar passages, not identical ones, so near-duplicate runbooks and sibling error codes collide in vector space; because top-k feeds the LLM, ranking matters as much as recall. The two channels fail in complementary ways, which is why hybrid exists. On the WANDS e-commerce benchmark, Denser AI citing Turnbull (2025) reports tuned hybrid NDCG of 0.7497 against pure vector 0.6953 and BM25 0.6983. Those figures are vendor-reported on that corpus; verify them on your own labels before you rely on them, and read the fusion walkthrough on hybrid search.

When the right passage never appears, you are looking at a retrieval failure. Start from wrong chunk, vocabulary mismatch, or missing document. When you need numbers rather than a symptom, measure with recall, precision, MRR and NDCG. IBM’s RAG topic page and Google Cloud’s overview both still introduce retrieval as the grounding step that keeps generation attached to your corpus; without that grounding, fluency is not evidence. The next question is how the scoring channels themselves differ before any transform is applied.

How do dense, sparse and hybrid scoring choose different chunks?

Dense, sparse and hybrid scoring choose different chunks because they measure different notions of match. Dense similarity ranks by embedding geometry: cosine, dot product or Euclidean distance between the query vector and each chunk vector. Sparse lexical scoring ranks by term statistics: how often query tokens appear, how rare those tokens are, and how long the document is relative to the collection. Hybrid runs both and fuses the lists so exact and semantic hits surface together.

Dense search is the default story in most RAG tutorials. The vectors themselves are produced by an embedding model; the metric that ranks them is a separate choice owned by similarity metrics. Cosine cancels length and keeps direction; the raw dot product keeps magnitude; Euclidean measures a straight-line gap. When embeddings are unit-normalised, cosine and the dot product rank identically, which is why many engines expose a fast inner-product path after a one-time normalise. Matching the metric to the embedding model’s training objective is the decision rule Weaviate’s distance-metrics guidance and Qdrant’s Essentials course both state (as of September 2026 for Weaviate docs). Google Cloud’s RAG overview and Meilisearch’s types catalogue both still introduce dense retrieval as the first mental model readers meet, which is why metric mismatch is a silent bug rather than an obvious configuration error.

Sparse search is the channel that still wins on identifiers. Weaviate runs keyword search as BM25F across the text properties it searches, with the same defaults major engines use: k1 of 1.2 and b of 0.75, matching Apache Lucene’s BM25Similarity, Elasticsearch’s similarity reference and Vespa’s BM25 docs. Weaviate’s keyword search documentation (as of September 2026) describes tokenization per property, English stopwords filtered at query time, and property boosting with a caret. BM25F scores are relative to the dataset and the query, so a fixed “relevant above X” threshold does not transfer across collections. That formula depth, and the cases where lexical ranking scores zero on synonyms, live on the lexical leaf rather than in this overview.

Hybrid is the production default when queries mix natural language with exact tokens. Weaviate’s hybrid search documentation (as of September 2026) runs vector search and BM25F in parallel on a single query string, then fuses the lists. The alpha parameter sets the balance: 0 is pure keyword, 1 is pure vector, and 0.75 is the server default, which applies only when a request arrives with no alpha value because some client libraries send their own. Since Weaviate v1.24 the default fusion method is relativeScoreFusion, which normalises each result set before weighting; the older rankedFusion was the default up to v1.23. Weaviate’s own FIQA recall benchmark reported about a 6 percent recall lift for relative score fusion over ranked fusion. Vector stores that expose native hybrid include Weaviate, Qdrant, Pinecone and Azure AI Search; the fusion maths and RRF constant live on the hybrid leaf, not here. K2view’s practical RAG guide and IBM’s topic page both treat hybrid as the upgrade path once dense-only demos meet production identifiers, which matches the complementary-failure story above.

How many of those ranked chunks enter the prompt is its own lever. choosing top-k is the cut-off between recall and noise, and raising k forever is how mid-context burial and token cost climb without answer quality following. Scoring decides who could win; top-k decides who the model is allowed to see. Query transforms sit before all of that when the user’s words are the problem.

How do query transforms change what gets searched?

Query transforms change what gets searched by rewriting the question before any chunk is scored. Microsoft’s common RAG techniques post (February 2025) groups query rewriting, multi-query expansion and hypothetical-document embedding as the interventions that raise recall when the user’s phrasing does not match the corpus vocabulary. Databricks’ workflow article treats the same family as part of the online path once a naive single-string retrieve is not enough. Practitioner write-ups often collapse those steps into a single “better retrieval” tip; the useful split is which string is searched versus which score ranks the hits.

The transform family splits by what it changes. query rewriting replaces or expands the user’s words so the scorer sees synonyms, expanded entities or a cleaner intent string. hyde generates a hypothetical answer, embeds that answer, and searches answer-to-answer instead of question-to-answer when the question sits far from document style. multi-query and fusion asks several variants and merges the result lists, usually with reciprocal rank fusion, at the cost of extra latency. self-query retrieval lets the model emit a structured metadata filter plus a semantic query so date, product line or tenancy constraints are enforced as filters rather than hoped for in the embedding. query decomposition splits a multi-part question into sub-queries when one retrieve cannot cover every clause. step-back prompting lifts a concrete question to a higher-level abstraction, retrieves on that abstraction, then answers the original ask with broader context in hand.

Transforms do not replace scoring. A rewrite that still runs dense-only will miss the same error codes a raw question would miss; a HyDE answer that invents a wrong identifier can pull the wrong neighbour set. The honest order is: fix the query only when measurement shows vocabulary mismatch, keep hybrid when exact tokens mix with paraphrase, and measure recall at the depth the prompt actually reads. Once the transform and the scorer are chosen, the remaining map question is which named technique owns the rest of the cluster.

Which retrieval technique matches the problem you have?

The retrieval technique that matches the problem you have is the one whose failure mode you are actually seeing. Scoring leaves own how chunks get ranked. Transform leaves own what is searched before ranking. Context-assembly leaves own how retrieved text is packed for the model. Control leaves own when and where to retrieve. Depth lives on those pages; the hub only orients.

On scoring, open bm25 when exact tokens, identifiers and rare names dominate and dense neighbours look plausible but wrong. Open similarity metrics when the question is cosine versus dot product versus Euclidean, or when magnitude is leaking into ranks. Open hybrid search when natural language and identifiers share the same queries, and you need both channels fused with an explicit alpha and relative score fusion. Open choosing top-k when recall@small-k is low, precision collapses as k grows, or token cost is rising faster than answer quality. Open maximal marginal relevance when the top ranks are near-duplicates and diversity matters more than raw similarity.

On transforms and fusion variants, open query rewriting when user words systematically miss document vocabulary. Open hyde when questions and documents sit in different styles. Open multi-query and fusion when one phrasing is not enough and you can afford several retrieves. Open rag-fusion when multi-query results need a named fusion recipe rather than an ad-hoc merge. Open self-query retrieval when metadata filters should be derived from the question. Open query decomposition when the ask has several independent clauses. Open step-back prompting when the concrete question needs a more abstract neighbour set first. Microsoft’s techniques catalogue and Meilisearch’s type list both surface several of these names as “types of RAG”; under this map they are retrieval interventions on a shared index, not separate architectures.

On how retrieved text is assembled for the model, open parent-document retrieval when small chunks rank well but the generator needs the surrounding parent passage. Open sentence-window retrieval when the hit is a sentence and the window around it must expand at read time. Open auto-merging retrieval when hierarchical nodes should collapse into a coherent parent before the prompt. Open contextual retrieval when chunks need context prefixes so the embedding carries document location, not only local text. Open recursive retrieval when one retrieve yields references that must be followed into further documents. Those assembly patterns trade recall of a precise span for coherence of the passage the generator reads, which is why they sit after scoring rather than instead of it.

On control, open query routing when different indexes or tools should handle different question types. Open adaptive retrieval when the decision is whether to retrieve at all, or how aggressively, per query rather than with a fixed always-on retrieve. Those control leaves sit beside scoring; they do not replace the need to know whether BM25, dense or hybrid is the channel underneath. LangChain’s retrieval documentation frames routing and tool choice as orchestration around the retriever; the scorer still has to return the right neighbours once the route is chosen.

If the symptom is only “answers are wrong” and you have not measured yet, do not pick a transform first. Measure whether the gold chunk ever entered the retrieved set, then choose the leaf that matches the miss. Without that measurement, stacking another named technique only adds latency.

How do you measure retrieval before changing the scorer?

You measure retrieval before changing the scorer by scoring the retrieved set against labelled gold chunks, separately from generation quality. Weaviate’s evaluation guidance (as of September 2026) states that RAG needs both component-level and end-to-end checks: retrieval accuracy and relevance on one side, faithfulness and correctness on the other. Databricks’ FAQ treats poor retrieval as the mode you must isolate before you blame the model. AWS’s RAG overview similarly separates retrieve quality from the generate step that consumes whatever came back.

The meters that matter for this stage are Recall@k, Precision@k, MRR and NDCG at the depth the prompt actually reads. Worked formulas and tables live on the retrieval-metrics leaf; the hub rule is simpler. If Recall@k is low, the scorer, the transform, the chunker or the index is the problem, not the prompt. If Recall@k is high and answers are still wrong, the failure has moved to reranking, context packing or generation. Raising top-k without a curve is how teams invent a temporary fix that burns tokens: the top-k leaf owns that trade-off, and the latency failure mode owns the cost side when oversized windows become the habit.

A labelled set of real queries with gold chunk ids is enough to start. Hold the chunker and embedder fixed, sweep the retrieval change you are testing, and keep generation metrics as a second plot so a recall win that destroys faithfulness is visible. Hybrid with an explicit alpha, relative score fusion since Weaviate v1.24, and a reranker on a short list is the usual production shape once measurement shows dense-only or BM25-only is incomplete. Until those numbers move, stacking another transform only changes latency. The same discipline applies when you promote a blog heuristic such as “always retrieve ten chunks”: without a Recall@k plateau on your corpus, that integer is a guess, not a configuration. Wikipedia’s RAG entry and LangChain’s retrieval docs both still describe retrieval as the step that supplies non-parametric context; measurement is how you know that step actually supplied the right context.

Retrieval in RAG is the score, the optional transform, the fusion and the top-k cut-off that decide what the model is allowed to see. Dense similarity, BM25F lexical match and hybrid fusion fail in complementary ways; Weaviate’s hybrid defaults of alpha 0.75 and relativeScoreFusion since v1.24 are the facts already locked on the BM25 and hybrid leaves. Query transforms change the string that is searched; they do not repair a channel that cannot see identifiers or a cut-off that drops the gold chunk. Measure recall at prompt depth, open the leaf that matches the miss, and treat reranking as the next stage after the shortlist exists. Chunking, embeddings and indexing remain upstream owners; architectures and agentic patterns remain downstream owners of control flow. Vector stores such as Weaviate, Qdrant and Pinecone change what retrieve can do; they do not replace the need to know which scoring channel failed. The sibling pages own the depth; the borders of the retrieval job stay on this map.