Measuring Retrieval: Recall, Precision, MRR and NDCG

Retrieval metrics score the ranked list of chunks before generation, so you can tell whether the right passage was fetched and how high it ranked, independently of the written answer. They exist because a fluent answer can still be built from the wrong neighbours, and an awkward answer can still have had the right passage in the window. Pinecone’s production series “RAG Evaluation: Don’t let customers tell you first” (captured 17 September 2026) puts the same split in vendor terms: a RAG pipeline is only as performant as the retrieval phase is accurate. Weaviate’s retrieval-evaluation-metrics article (Leonie Monigatti, 28 May 2024) catalogues the family as Precision@K, Recall@K, MAP@K, MRR@K, and NDCG@K, and splits them into metrics that only count relevant items in the top K and metrics that also care about position. The method is to work those formulas on one published list, then say which meter to read for which job.

What do retrieval metrics measure?

Retrieval metrics measure whether the right chunks appear in the top-k and how they are ordered. They measure nothing about the generated answer. Every score needs a labelled set of relevant passages for each query, from human labels or a trusted gold set, because Precision@k and Recall@k are comparisons against that set, not against the model’s fluency. In RAG, k is usually the number of chunks you actually put in the model’s context window, so the metric grades the same cut-off the generator sees. A metric computed at k=100 while the prompt only reads five chunks is answering a different question than the pipeline asks.

Pinecone’s chapter groups the family first by how relevance is labelled. Binary relevance treats a result as relevant or not. Graded relevance assigns shades, for example 0 to 5, when one passage is a complete answer and another is only related. Inside the binary group it then splits order-unaware metrics (Precision@k, Recall@k, F1@k), which ignore position inside the window, from order-aware metrics (MRR, Average Precision), which penalise a relevant hit that arrives late. Weaviate draws the same line with different words: Precision and Recall are not rank-aware; MAP, MRR, and NDCG are. Both splits matter in RAG because the generator does not read an unordered bag. It reads a list, and Liu et al., “Lost in the Middle” (Transactions of the Association for Computational Linguistics, 2024), document that mid-list evidence is the evidence most likely to be ignored.

mbrenndoerfer’s RAG evaluation guide (captured 17 September 2026) treats that retrieval-versus-generation split as the first design choice: retrieval metrics tell you whether the context window was usable; generation metrics tell you what the model did with it. Redis’s RAG-system-evaluation post (Rini Vasan, 13 January 2026) gives the diagnostic reason in production language: if you only score the final answer, you cannot tell whether retrieval grabbed irrelevant documents, the model hallucinated despite good context, or ranking buried the useful chunk at a position the prompt never emphasises. The labelled set that makes those retrieval scores computable is building a golden test set. The wider family list, including generation and context scores, sits with RAG evaluation metrics. How those scores become a launch decision is the complete method, tool-neutral. What the formulas actually return, on one list, is the next question, because a definition without a number is still a slogan.

How do you calculate Precision@k and Recall@k?

Precision@k is the fraction of the top-k results that are relevant. Recall@k is the fraction of all known relevant items that appear anywhere in that top-k. Weaviate writes Precision@K as (number of relevant items in K) / K, and Recall@K as (number of relevant items in K) / (total number of relevant items). Hit@k is 1 if at least one relevant item appears, else 0: a binary recall floor that does not care how many relevant items exist beyond the first. Pinecone adds F1@k as the harmonic mean of precision and recall at the same k, for the case where missing a relevant document and retrieving too many irrelevant ones are both costly.

LangCopilot’s single-query example (“RAG Evaluation Metrics Explained: Recall@K, MRR, Faithfulness”, 17 September 2025) is the list most RAG metric posts reuse. Relevant ground truth: {doc3, doc5, doc10}. Top-5 retrieved: [doc2, doc5, doc3, doc8, doc7]. Two of those five are relevant, so Precision@5 is 2 / 5 = 0.40. Two of the three known relevant documents appear, so Recall@5 is 2 / 3 = 0.67. Hit@5 is 1, because at least one relevant item is present. doc10 never entered the window. The arithmetic is LangCopilot’s published expected output, recomputed on 17 September 2026 from that same list.

Weaviate’s pastry query is a second sourced ranking, not a restatement of LangCopilot. The published qrel marks donut, muffin, and scone as relevant for “sweet pastry”. The published run returns donut (0.95), bagel (0.9), muffin (0.8), croissant (0.7). pytrec_eval on that pair reports Precision@4 = 0.5 (donut and muffin of four) and Recall@4 = 0.666…, which is 2 / 3: scone never appeared. Two lists, two corpora, the same two fractions. High recall with low precision means the window is noisy: the right chunk may sit beside junk, which is why RAG returns the wrong chunk. High precision with low recall means a clean window that still missed evidence, which is when RAG misses the document. Pinecone’s recall@k walk-through makes the k-window concrete: if three documents are relevant and k=1, recall is 1 / 3 even when that one hit is perfect, because two relevant items still sit outside the window. Average these metrics over many queries. A single query is for learning the arithmetic. Rank of the first relevant hit is a different number, and that is the next meter.

How does MRR score a ranking?

Mean Reciprocal Rank scores a ranking by averaging, over queries, one over the rank of the first relevant item. Weaviate writes MRR@K as the mean of those reciprocal ranks across the evaluated queries or users, and writes the per-query Reciprocal Rank as 1 / (rank of the first relevant item). On LangCopilot’s list, the first relevant hit is doc5 at rank 2, so the reciprocal rank is 1 / 2 = 0.50, matching LangCopilot’s expected MRR for this data. A first-relevant at rank 1 scores 1.0. A first-relevant at rank 5 scores 0.2. If nothing relevant appears in the cutoff, the reciprocal rank for that query is 0.

Pinecone treats MRR as the order-aware metric you want when the application only needs one correct answer near the top: fast decision-making, a single tool to call, a single citation to show. Redis gives a production reading of the same scale: an MRR above 0.8, in Redis’s wording, indicates most queries return a relevant result in the top one or two positions, with the caveat that actual targets vary by domain and query complexity. That 0.8 figure is Redis’s stated indication, not a universal launch bar. Weaviate notes that MRR is not available in the pytrec_eval package as of that May 2024 writing, so teams implementing Weaviate’s pastry example in pytrec_eval have to compute reciprocal rank themselves.

MRR’s blind spot is structural: it ignores every relevant item after the first. A ranking that puts one correct chunk at rank 1 and noise in ranks 2 through 10 still scores 1.0 for that query, even when the rest of the window is useless to a generator that reads five chunks. Weaviate is explicit that the metric “only considers the order of the first relevant results but not the number or order of the other relevant results.” On the LangCopilot list that limitation is visible in the arithmetic: MRR is 0.50 because doc5 landed at rank 2, and doc3 at rank 3 contributes nothing more. For multi-chunk RAG context, that is a feature only if the product truly consumes the top hit. If the product stuffs k chunks into a prompt, you need a metric that sees the rest of the list. MAP is the binary metric built for that job.

How does MAP differ from MRR?

Mean Average Precision differs from MRR because it scores every relevant item in the top K, not only the first, and it rewards putting those relevant items earlier. Weaviate defines MAP@K as the mean, over queries, of Average Precision@K. Average Precision@K itself sums Precision(k) at each rank k where the item is relevant, then divides by the number of relevant items N. Pinecone’s Average Precision walk-through is the same idea with a sliding Precision@k: relevant results placed higher improve the average; relevant results placed later drag it down. MRR would have stopped at the first green light. MAP keeps scoring.

Weaviate’s published pastry MAP uses a slightly tighter qrel than the Precision@4 example: donut and muffin marked relevant, run still donut 0.95, bagel 0.9, muffin 0.8, croissant 0.7. pytrec_eval reports map_cut_4 = 0.833…. The 0.833 is Average Precision on that one query, and Weaviate notes that pytrec_eval does not then average across multiple queries; the “mean” in MAP is your job once you have more than one query. On LangCopilot’s list, Average Precision at 5 is (Precision@2 × 1 + Precision@3 × 1) / 3, because doc5 is relevant at rank 2 (precision 1/2) and doc3 is relevant at rank 3 (precision 2/3), and three relevant documents exist in the qrel. That is (0.5 + 0.667) / 3 = 0.39 when N is all three relevant items including the missing doc10. The missing relevant item is why MAP sits below MRR on the same list: MRR never penalises the document you never retrieved.

Weaviate also records that MAP@K is the default metric on the Massive Text Embedding Benchmark (MTEB) leaderboard for the Reranking category. That default is a statement about how embedding rerankers are compared in public, not a statement that MAP is the right RAG dashboard number. For a RAG chat stack that reads several chunks, MAP is the binary, multi-relevant counterpart to MRR: use it when several documents can be relevant and order among them matters, and you do not yet have graded labels. Once judges can say “exactly the answer” versus “related”, NDCG is the metric that can use those grades.

How does NDCG score a ranking?

Normalised Discounted Cumulative Gain scores a ranking by summing a discounted gain at each position, then dividing by the score an ideal ranking would have received at the same k. Weaviate writes NDCG@K = DCG@K / IDCG@K, and writes DCG@K as the sum of rel_i / log2(i+1). Towards Data Science Part 3 of “How to Evaluate Retrieval Quality in RAG Pipelines” (Eleni, captured 17 September 2026) uses the same linear gain in its Python: rel / math.log2(i + 1), then IDCG as DCG of the relevance list sorted descending. Pinecone’s graded-relevance section is the reason the numerator is allowed to be 2 or 5 rather than 1: a result can be extremely relevant while another is only somewhat relevant, and NDCG is the family that can see the difference.

LangCopilot’s post stops at Precision, Recall, and MRR for its five-document list. Extending that list with binary NDCG@5 is the extra measurement. Relevances for the five ranks are [0, 1, 1, 0, 0]. On binary labels, the exponential gain 2^rel − 1 equals the linear gain rel, so the two common DCG conventions coincide. DCG = 1/log2(3) + 1/log2(4) = 1.1309. IDCG for three relevant documents in a top-5, ideal [1, 1, 1, 0, 0], is 2.1309. NDCG@5 = 1.1309 / 2.1309 = 0.53, computed 17 September 2026 from LangCopilot’s published ranking. The 0.53 is not a LangCopilot figure. LangCopilot did not publish it.

Weaviate’s jam query is the graded case the binary list cannot show. The published qrel for “goes well with jam” marks bagel, croissant, and roll at grade 2, and scone, muffin, and donut at grade 1. The published run returns pretzel 0.9, bagel 0.85, muffin 0.7, donut 0.6. pytrec_eval reports ndcg_cut_4 = 0.467239. Recomputed 17 September 2026, that published figure rounds to 0.4672. On the same top-4, treating any grade above 0 as relevant, Precision@4 is 3 / 4 = 0.75 (bagel, muffin, donut; pretzel is not in the qrel), Recall@4 is 3 / 6 = 0.5, and the first relevant item is bagel at rank 2 so reciprocal rank is 0.5. Binary metrics call that ranking mostly fine. Graded NDCG does not, because pretzel occupied rank 1, croissant and roll never appeared, and muffin and donut are only grade 1. Towards Data Science’s warning about raw DCG is the reason for the normalisation: DCG grows as k grows whenever extra items have any gain, so two windows of different length cannot be compared on DCG alone. NDCG puts both on a 0 to 1 scale against their own ideal. Weaviate records that NDCG@K is the default metric on the MTEB leaderboard for the Retrieval category, the counterpart to MAP’s reranking default. Use NDCG when order inside the window matters, especially after a second-stage model reorders what retrieval returned, or when judges give graded relevance instead of a binary yes or no.

Which retrieval metric should you use?

The retrieval metric you should use is the one that matches what the application actually reads from the ranked list, not the one that is fashionable on a leaderboard. Recall@k asks whether the answer ever entered the context window, which is the default RAG sanity check: if the passage is absent, the generator cannot cite it. Precision@k asks whether that window is mostly on-topic, or whether the model is drowning in neighbours. MRR asks how fast the first relevant hit arrives, which matches top-1 jobs such as routing a tool. MAP asks whether several relevant items are packed toward the top of a binary list. NDCG asks whether the whole window is close to the ideal order, including graded labels.

Weaviate’s summary of popularity is a map of public benchmarks, not of product dashboards: MAP@K is most popular for evaluating recommendation systems, NDCG@K is highly popular for evaluating retrieval systems, and those two defaults are also how MTEB splits reranking versus retrieval. Pinecone’s “good enough” chapter is the product counterpart: Precision@k is the metric when reviewing an irrelevant result is expensive or first impressions matter; Recall@k is the metric when missing a relevant document is the expensive failure; F1@k is the compromise; MRR is the metric when one correct answer at the top is the job. edge-ai-vision’s February 2026 practical guide frames the same three names, recall, precision, and NDCG, as the working set a ranking system actually ships with. None of those sources publish a single RAG threshold that transfers across corpora.

For a typical RAG chat stack, start with Recall@k and Precision@k on a labelled set, then add NDCG when you introduce a reranker or graded labels. Add MRR only if some path in the product consumes a single top hit. Hybrid lexical plus dense retrieval changes these scores because BM25 and a dense index fail on different queries, so the fused list has to be measured as its own retriever; that fusion is Combining BM25 and Vector Retrieval. A low score still has to be read as a failure class, or the metric becomes a vanity number.

What does a low retrieval score mean?

A low retrieval score points at a retrieval failure, not a generator failure, unless every retrieval meter is fine and the answer is still wrong. Redis’s cascade is the reason to keep the stages apart: if the retriever misses critical documents or buries them at position 20, the language model never sees them, the answer looks like a hallucination, and generation metrics alone will not tell you why. Pinecone’s opening claim is the same constraint in one sentence: retrieval-augmented generation is only as performant as retrieval is accurate. The diagnosis is which meter moved.

Low Recall@k or Hit@k means the relevant passage never made the window. That is a coverage failure: the document is absent from the index, the query did not match it, or k is too small, which is missing the obvious document. Low Precision@k means the top-k is mostly the wrong neighbours, so the generator is reading on-topic-looking junk, which is wrong chunk retrieved. Acceptable recall with low MRR, MAP, or NDCG means the right document is present but buried: the first relevant hit is late, or the graded best hit is not at rank 1. Once that long context reaches the model, check when the right chunk is ignored, because a chunk at position 8 of 10 can score as retrieved and still never be used. Redis’s own illustration of the bury-case is a retriever at 0.85 precision that still puts critical documents at position 15, where the model never sees them; that 0.85 is Redis’s example, not a published benchmark of a named system.

When every retrieval meter is high and answers are still wrong, leave retrieval. The index can still hold a policy you already deleted, which is why RAG answers from documents you already deleted. The generator can still ignore good context or invent a clause, which is what measuring generation is for. Retrieval metrics do not grade freshness, permissions, or faithfulness. They grade whether the retrieval step put the right text in the window, and how high it sat.

Retrieval-augmented generation is a retrieve-then-generate pipeline, and the retrieval half is still information retrieval: Precision@k and Recall@k count what entered the window, MRR scores the first relevant hit, MAP scores every binary relevant hit by position, and NDCG scores the whole window against an ideal, including grades. LangCopilot’s five-document list returns Precision@5 of 0.40, Recall@5 of 0.67, MRR of 0.50, and, extended here, NDCG@5 of 0.53. Weaviate’s pastry and jam queries add a second corpus where Precision@4 is 0.5, Recall@4 is two thirds, MAP@4 is 0.833, and graded NDCG@4 is 0.4672. Pick the meter that matches what the prompt actually reads, then treat a low score as a named failure rather than as a reason to change the generator first.