Retrieval in RAG: Getting the Right Context to the Model
The step the whole architecture is named after. What is scored, what is returned, and what determines whether the answer can be right.
Retrieval in RAG scores indexed chunks against the query and returns what the model sees — if the right passage is missing, the answer cannot be right. Pick the scoring method or query transform, then open its leaf.
How does retrieval work in RAG?
Retrieval in RAG takes the user query, scores indexed chunks — by dense similarity, sparse lexical match, or both — ranks them, and returns a top-k set that becomes the model’s context. LangChain’s retrieval docs frame that as fetching relevant external knowledge at query time; AWS’s RAG overview lists the same beat as retrieve-then-augment: pull relevant passages, then put them in the prompt. The stages, in order:
- Optional query transform — rewrite, expand, invent a hypothetical answer (HyDE), fan out multi-query variants, or let the model emit a structured filter (self-query) before anything is scored.
- Score — dense approximate nearest-neighbour search over embeddings, and/or BM25 over terms. Dense finds paraphrases; BM25 finds exact tokens.
- Fuse (when both run) — merge the two ranked lists, typically with Reciprocal Rank Fusion. The full fusion maths live on hybrid search.
- Select top-k — keep the shortlist the model will actually see. How large k should be is its own decision on choosing top-k.
- Hand off — the selected chunks become context for generation. A second-stage cross-encoder that reorders that shortlist is reranking, not retrieval itself.
What gets scored and what gets returned are the only decisions that matter at this stage. Everything upstream (chunking, embeddings) sets the ceiling; everything downstream (rerank, generate) can only work with what retrieval handed over.
Why does retrieval quality determine the answer?
Retrieval quality determines the answer because the model only sees the chunks that came back — a faithful answer to the wrong context is still wrong. Nerd Level Tech’s RAG systems guide puts it directly: retrieval quality determines answer quality. InfoQ’s hybrid-retrieval article (June 2026) makes the mechanism concrete: embeddings find similar passages, not identical ones, so near-duplicate runbooks and sibling error codes collide in vector space; because top-k feeds the LLM, ranking matters as much as recall.
Dense-only search fails where exact tokens matter. Product codes, error numbers, statute IDs and rare names are the cases Denser AI’s hybrid guide (2026) lists as the dense blind spot — BM25 catches the literal string; the embedding often does not. The two channels fail in complementary ways, which is why hybrid exists. On the WANDS e-commerce benchmark, Denser AI citing Turnbull (2025) reports tuned hybrid NDCG of 0.7497 against pure vector 0.6953 and BM25 0.6983 — verify those figures on your own corpus before you rely on them; the fusion walkthrough is on hybrid search.
When the right passage never appears, you are looking at a retrieval failure — start from wrong chunk, vocabulary mismatch, or missing document. When you need numbers rather than a symptom, measure with recall, precision, MRR and NDCG.
What kind of retrieval do you need? Find the technique
Match the question you are actually asking to a technique below — scoring first if you are choosing how chunks get ranked, query transforms if the user’s words are the problem. Depth lives on those pages; this hub only orients.
Scoring — how chunks get ranked
These four leaves own the scoring decision: what similarity means, how many chunks come back, and whether lexical and dense signals run alone or together.
Query transforms — change the query before you search
These four leaves own what happens to the question before any chunk is scored — rewrite it, invent an answer to embed, fan out variants, or emit a metadata filter.
Don’t know where to start?
If your queries mix natural language with identifiers, error codes or product names, start with hybrid search. If you only know answers are wrong, measure whether the right chunk was ever retrieved — retrieval metrics first →
What is retrieval in RAG?
Retrieval is the step that scores indexed chunks against the user query and returns a top-k set of passages for the model to use as context. If the right passage is not in that set, the answer cannot be right — generation only sees what retrieval handed over.
How does retrieval work in RAG?
Optionally transform the query first (rewrite, HyDE, multi-query, or a self-query filter), then score chunks with dense similarity and/or BM25, fuse the lists if both channels run, select top-k, and pass those passages into the prompt. Reranking is a separate stage that reorders an already-retrieved shortlist.
What is hybrid search in RAG?
Hybrid search runs BM25 and dense vector retrieval in parallel, then fuses both ranked lists — usually with Reciprocal Rank Fusion — so exact tokens and semantic neighbours surface together. It is the default upgrade when embeddings alone miss identifiers, error codes and rare terms.
Why does vector search miss exact terms?
Embeddings score semantic similarity, not string identity. Product codes, error numbers and rare names often land in weak or crowded regions of embedding space, so a nearby paraphrase outranks the line that literally contains the token. BM25 catches those exact strings; hybrid keeps both channels.