RAG Evaluation Metrics: The Complete Catalogue

RAG evaluation metrics are measurements that score retrieval quality, generation quality, or end-to-end context use separately, so you can tell which stage of the pipeline failed. A single accuracy number cannot do that job, because a faithful answer from the wrong passage, a relevant answer that invents a clause, and a complete miss of the document all look like “wrong” from the outside. Jeffrey Ip’s DeepEval guide (Confident AI, 10 October 2025) states the reason as a product, not a sum: if either the retriever or the generator fails, overall quality can drop to zero. Yu, Gan, Zhang, Tong, Liu and Liu’s survey “Evaluation of Retrieval-Augmented Generation” (arXiv 2405.07437v2, 3 July 2024) splits the same work into retrieval, generation, and the system as a whole. The families in common use, what each one can see, and which failure it can actually diagnose, follow in query order.

What are RAG evaluation metrics?

RAG evaluation metrics quantify how well a retrieval-augmented generation pipeline retrieves relevant context and generates a grounded answer. They are a panel, not a single accuracy number. Confident AI’s guide defines RAG evaluation as testing the retriever and the generator separately with five industry-standard scores: answer relevancy, faithfulness, contextual relevancy, contextual recall, and contextual precision. Redis’s RAG-system-evaluation post (Rini Vasan, 13 January 2026) compresses the same panel into three dimensions: context relevance (sometimes called context precision), groundedness (also called faithfulness), and answer relevance. Patronus lists five names that map onto those dimensions: context relevance, context sufficiency, answer relevance, answer correctness, and answer hallucination. The labels differ. The stage split does not.

The split exists because generation metrics grade the answer against retrieved context, so they cannot detect a wrong chunk that was answered faithfully. Retrieval metrics grade the ranked list against labelled relevant documents, so they cannot detect a generator that ignored a good list. IBM’s RAG cookbook (Result Evaluation, updated 15 November 2024) puts the same hybrid-structure problem in lifecycle terms: evaluation has to be reproducible in the experiment-and-tuning phase and in the monitoring phase, and the effort to build a golden dataset of reference answers and reference contexts should not be underestimated. Yu et al. (2024) add that isolated component scores still miss the interplay, latency, ambiguous queries, and a changing corpus, which is why a catalogue has to say what each metric cannot see as well as what it measures.

Weaviate’s retrieval-evaluation-metrics article (Leonie Monigatti, 28 May 2024) is the retrieval half of that panel in established information-retrieval language: Precision@K, Recall@K, MAP@K, MRR@K, and NDCG@K, all evaluated on the top K results, all on a 0 to 1 scale where higher is better. Those formulas, and the pastry and jam worked lists, live with measuring retrieval. The generation half, faithfulness and answer relevance as Ragas named them in Weaviate’s companion RAG-evaluation post, lives with measuring generation. How you run the whole method, labelled queries then judges then production, is tool-neutral evaluation. The first family most teams under-build is retrieval, because it is the older information-retrieval problem sitting under a new generator.

Which metrics measure RAG retrieval?

Retrieval metrics measure whether the right chunks appear in the top-k and how they are ordered, against a labelled set of relevant documents for each query. Evidently AI’s RAG evaluation guide (captured 17 September 2026) lists the ground-truth set: Precision@k (of the top k items, how many are relevant), Recall@k (of all relevant items, how many appeared in the top k), Hit Rate (whether at least one relevant item appeared), and NDCG@k (a rank-aware score that rewards relevant items appearing higher). Weaviate walks the same family and adds MAP@K and MRR@K. Redis illustrates Precision@K with a toy fraction: if you retrieve 10 chunks and 7 are relevant, precision is 0.7. That 0.7 is an arithmetic example, not a published system score.

The one-line definitions, without the worked arithmetic, are these. Precision@k is relevant-in-window divided by k. Recall@k is relevant-in-window divided by all known relevant items. Hit@k is 1 if any relevant item appears. MRR is the mean of 1 / rank of the first relevant item. MAP is the mean of Average Precision, which scores every relevant item by the precision at its rank. NDCG discounts relevant hits by log position and divides by the ideal ranking. Pinecone’s RAG evaluation chapter (captured 17 September 2026) also names F1@k as the harmonic mean of precision and recall when missing a relevant document and retrieving too many irrelevant ones are both costly. k should match the number of chunks the prompt actually reads. A Recall@100 on a five-chunk prompt is a different experiment.

GeeksforGeeks groups those names as retrieval-level metrics and notes the limitation in its comparative table: they are simple and interpretable, and they do not evaluate answer quality, fluency, or coherence. That limitation is the point of the catalogue. Retrieval scores tell you whether the retrieval step put usable text in the window. They do not tell you whether the generator used it. The labelled set they require is building a golden test set. Formulas and scored examples, including LangCopilot’s Precision@5 of 0.40 and Weaviate’s pastry Precision@4 of 0.5, belong with measuring retrieval. What those scores hide, even when the counts look healthy, is whether order inside the window was any good, which is a different family of meters.

How do rank-aware metrics differ from precision and recall?

Rank-aware metrics differ from precision and recall because they score position, not only count. Weaviate states the split directly: Precision and Recall are not rank-aware and only reflect the number of relevant items in the top K; MAP, MRR, and NDCG are rank-aware and consider number and position. Pinecone draws the same line with different words. Order-unaware metrics (Precision@k, Recall@k, F1@k) ignore which order relevant items arrived in. Order-aware metrics (MRR, Average Precision) penalise a relevant hit that arrives late. IBM’s cookbook calls rank-aware metrics from recommendation systems appropriate for RAG, and lists MRR, NDCG, and MAP as the retrieval set, with the explicit limitation that MRR does not penalise a retriever for assigning a low rank to the other ground-truth documents.

The practical difference in a RAG prompt is easy to state and easy to miss on a dashboard. Two windows can both contain two relevant chunks out of five, so Precision@5 is 0.40 either way. If those two chunks occupy ranks 1 and 2, the generator reads them first. If they occupy ranks 4 and 5, the same precision number describes a window the model may never emphasise. Redis’s production illustration is a retriever at 0.85 precision that still buries critical documents at position 15, where the language model never sees them. That 0.85 is Redis’s example, not a benchmark of a named system, and it is the reason precision without a rank-aware companion is a false comfort. Liu et al., “Lost in the Middle” (Transactions of the Association for Computational Linguistics, 2024), document the generation-side of the same geometry: mid-list evidence is the evidence most likely to be ignored once it does reach the prompt.

Weaviate also records where the public embedding leaderboard landed on the split. MAP@K is the default metric on the Massive Text Embedding Benchmark (MTEB) leaderboard for the Reranking category. NDCG@K is the default for the Retrieval category. NDCG is the one that needs graded relevance (relevant versus somewhat relevant versus not) rather than a binary label; binary labels are cheaper and more common; graded labels are what NDCG is for. MAP stays in the binary, multi-relevant world. MRR stays in the first-hit world. Precision and Recall stay in the count world. A RAG chat stack that stuffs several chunks into a prompt needs at least one count metric and one rank-aware metric, or a clean precision number will hide a buried hit. Which generation scores then grade what the model wrote from that window is the next family.

Which metrics measure RAG generation?

Generation metrics measure what the language model wrote against the retrieved context and against the question, not against the model’s general knowledge. Confident AI assigns two generator metrics: faithfulness (whether the response contains hallucinations relative to the retrieval context) and answer relevancy (how relevant the generated response is to the input). Redis computes faithfulness by extracting statements from the answer, classifying each as supported or unsupported by context, then dividing supported statements by total statements. Patronus calls the unsupported share answer hallucination, “also known as faithfulness,” and adds answer correctness as alignment with a gold response. IBM lists faithfulness, robustness (insensitivity to input perturbations such as whitespace and case), ROUGE, and BLEU on the generation side. The names proliferate. The jobs do not: stay inside the context, address the question, and, when a reference exists, match it.

Groundedness is the same job as faithfulness under another stack’s label. IBM’s cookbook, describing LLM-as-a-judge properties, names groundedness as whether generated output is grounded in facts present in the input context. Redis says groundedness is also called faithfulness. Microsoft-oriented guides often prefer groundedness; Ragas, as summarised in Weaviate’s RAG-evaluation post, prefers faithfulness and answer relevancy. An answer can score high on faithfulness and low on relevancy: copying the context verbatim is faithful and not an answer. An answer can score high on relevancy and low on faithfulness: a fluent on-topic paragraph that invents a tenure rule. Redis states production targets in its own voice: high-stakes apps (medical, legal, financial) need faithfulness above 0.9, and standard production systems should maintain above 0.8. Those thresholds are Redis’s stated bars, not a cross-corpus law, and Redis notes that LLM-based evaluation can introduce its own biases.

Two tools reporting “faithfulness” do not automatically agree, because the prompt, the judge model, and the definition of a supported claim differ. That disagreement, and the prompts that produce it, belong with measuring generation. Robustness, in IBM’s definition, is a different generation concern: whether the solution adapts to input variations. It is not a substitute for faithfulness. GeeksforGeeks groups answer relevance, groundedness, hallucination rate, and coherence as end-to-end metrics that are harder to compute automatically and may require human evaluation. The packaged implementations of those scores, Ragas, DeepEval, TruLens, Phoenix, LangSmith, are compared at RAG evaluation tools. Ragas’s own assumptions sit at Ragas explained. A generation score that looks healthy still depends on whether the retrieved set contained the right facts, which is the context family sitting between the two stages.

Which metrics measure context use end to end?

Context metrics measure the retrieved set as a whole for the question, often with an LLM judge, rather than comparing chunk IDs to a pre-labelled relevant set. Confident AI’s five-metric panel gives three of those names to the retriever: contextual relevancy (how relevant the retrieval context is to the input, used to judge top-K and chunk size), contextual recall (whether the retrieval context contains all the information required to produce the ideal output), and contextual precision (whether the retrieval context is ranked in the correct order, higher relevancy first). Redis’s generation-quality section treats context precision and context recall as the bridge between retrieval and generation: precision asks whether relevant chunks appear early rather than scattered; recall asks whether all necessary information appears in the retrieved chunks. Redis’s own warning: when context recall drops below 0.8, the model does not have necessary information and might fabricate missing details. That 0.8 is Redis’s stated drop-line, not a published leaderboard cutoff.

Patronus uses academic aliases for the same two jobs. Context relevance is “also known as context precision” and is computed by extracting statements in the retrieved context and classifying each as relevant or irrelevant with another model call. Context sufficiency is “similar to context recall” and requires a gold answer: it counts how many statements in the gold reference can be matched in the retrieved context. High faithfulness with low context recall is the diagnostic pattern the bridge exists to catch. The model stayed faithful to incomplete context, so the answer is locally true and globally wrong. Low contextual precision with high recall is the noisy-window pattern: the facts were retrieved, then buried or mixed with neighbours, which is how a rank-aware retrieval metric and a context-precision judge can agree without being the same formula.

LangSmith’s evaluate-RAG tutorial (LangChain docs, captured 17 September 2026) implements the bridge as four evaluators on one dataset: correctness of the response against a reference answer, relevance of the response to the input, groundedness of the response against retrieved docs, and relevance of retrieved docs to the input. That is generation, generation, generation, and context, with no labelled chunk IDs required. Evidently AI’s LLM-judged relevance path is the cheaper alternative to a ground-truth retrieval benchmark: you can score each chunk or the bundled context, and you cannot compute classical Recall@k unless you also defined what should have been retrieved. Those judges are a measurement instrument with position bias, verbosity bias, and self-preference, which is the depth owned by LLM-as-a-judge. Classic overlap metrics try to skip the judge entirely, and they fail at a different job.

Do BLEU and ROUGE work for RAG evaluation?

BLEU and ROUGE do not work as RAG faithfulness metrics, because they measure n-gram overlap with a reference, not whether an answer is supported by retrieved context. Elastic’s search-labs post (Quentin Herreros, Thomas Veasey, and Thanos Papaoikonomou, 1 December 2023) is the ranking page that treats this family in depth. BLEU is Bilingual Evaluation Understudy, a 0 to 1 precision-of-n-grams score against one or more references, originally for translation. ROUGE is Recall-Oriented Understudy for Gisting Evaluation, recall of shared words or phrases, originally for summarisation. METEOR adds a harmonic mean of precision and recall, plus synonyms, stemming, and a fragmentation penalty, and Elastic notes it correlates more strongly with human judgements than BLEU. Patronus groups BLEU and ROUGE as statistical metrics that “do not consider the meaning of the outputs” and are “not recommended anymore” for LLM-generated text. IBM still lists both under generation metrics, with the same cons: they penalise legitimate paraphrases and miss subtle hallucinations.

Elastic then walks the next two generations of overlap. Perplexity measures how confidently a model predicts a token sequence and requires the model’s own probabilities, so it cannot compare two systems with different vocabularies or context lengths, and it is not a hallucination detector. BERTScore, BLEURT, and BARTScore move the comparison into embeddings or a critic model. BERTScore (Zhang et al., arXiv 1904.09675) compares contextual token embeddings. BLEURT (Sellam et al., arXiv 2004.04696) adds pre-training on randomised Wikipedia perturbations and fine-tuning on human ratings. BARTScore (Yuan et al., arXiv 2106.11520) treats evaluation as generation probability under a BART critic. Elastic’s judgement is that these are still similarity or perplexity in nicer clothes, slower than BLEU, and still popular as a Hugging Face leaderboard proxy that is not sufficient for current LLMs. GeeksforGeeks’s comparative table puts BLEU, ROUGE, METEOR, BERTScore, and perplexity in one generation-metrics row whose weakness is “may miss semantic meaning, context or factual correctness.”

UniEval is Elastic’s stated choice for evaluating RAG, not a claim that it is the field’s winner. Zhong et al. (arXiv 2210.07197) unify evaluation dimensions as boolean question answering on a T5 model. Elastic describes an 800-million-parameter UniEval trained on coherence, consistency, fluency, and relevance, and treats RAG question answering as aggressive summarisation when parametric memory is avoided. Elastic compared that 800-million-parameter T5 with a GPT-4-based G-Eval judge they estimated at 1.76 trillion parameters, and judged that any correlation advantage was not worth the cost; those parameter counts and that cost judgement are Elastic’s, dated 1 December 2023. In their own experiment they scored Mistral-7b-Instruct and Falcon-7b-Instruct with UniEval on 200 queries from 18 datasets (the figure they caption as 3600 queries) and reported Mistral higher on all four dimensions. Consistency, in a second Elastic experiment on about 100 SQuAD 2.0 queries, tracked hallucination probability without being a silver bullet: parametric-memory answers that were factually correct still looked like consistency failures because they did not come from the prompt. The generation metrics that actually grade context support, rather than n-gram overlap, remain faithfulness and groundedness. When those automatic scores still miss tone, safety, or a subtle legal hedge, a rater has to look.

When does human evaluation still belong in a RAG metric panel?

Human evaluation still belongs in a RAG metric panel when the property you care about is not a chunk-ID match or an n-gram overlap: relevance as a reader experiences it, informativeness, factual accuracy against the world, and clarity. GeeksforGeeks lists those four criteria, plus a fifth row that is really a method list: rating scales, pairwise comparisons, and expert reviews. Rating scales score responses on relevance, accuracy, and clarity. Pairwise comparison asks which of two responses is better. Expert review asks a subject-matter rater to assess quality, factual correctness, and usefulness. The comparative table in that same article is honest about the trade: human evaluation captures nuance, context, readability, and factual correctness, and it is time-consuming, subjective, and not easily scalable.

Evidently AI’s manual relevance labeling is the retrieval form of the same work. Instead of predefining relevant document IDs, you let the system retrieve, then assign relevant, partially relevant, or not relevant to what came back. Evidently notes that Google does this with internal rater guidelines, that the method cannot compute recall because it never defined what should have been retrieved, and that it does not scale without a review team. Redis keeps humans in the loop for a different job: automated metrics make continuous production monitoring work, and “human review still validates edge cases and checks LLM-as-judge reliability.” IBM’s cookbook treats LLM-as-a-judge as the technique that emerged to overcome the cost of a reference-based engine, with decent correlation with human judgement on properties such as safety, groundedness, sentiment, toxicity, and language style, which is an argument for judges as a scalable proxy, not as a replacement for a rater on launch-critical answers.

The practical rule is smaller than a methodology paper. Use labelled chunk IDs when you can afford them and you need Recall@k. Use an LLM judge when you need to score production volume and you are willing to spot-check the judge. Use a human rater on a sample before a launch that quotes policy, and on the disagreements between two automatic scores. Patronus’s best-practice list starts with establishing a gold standard early, then choosing accuracy metrics from the use case, then security metrics, automated pipelines, evolving gold references, and drift thresholds. The gold standard in that list is still a human artefact. The sample a rater sees should include the queries automatic metrics painted green, because those are the ones a dashboard will not escalate. Which green score maps to which failure is the diagnostic table the rest of the catalogue is for.

Which RAG metric diagnoses which failure?

Each RAG metric can only see one failure class, so the metric you read has to match the symptom you actually have. Confident AI splits retriever failures (uninformative embeddings, poor chunking, weak reranking, suboptimal top-K) from generator failures (ignores key information, focuses on the wrong details, misreads the prompt, weak model). Redis’s warning is the same point in production language: miss any of context relevance, groundedness, and answer relevance, and you are optimizing the wrong bottleneck. IBM’s cookbook turns the pattern into a repair list: if retrieval metrics are weak and generation metrics look fine, adjust chunking, clean the corpus, add metadata, rerank, or rephrase the query; if retrieval looks strong and generation looks weak, change the prompt, the decoding strategy, or the model. GeeksforGeeks’s comparative table names strengths and weaknesses per family and still does not route a low score to a named failure. A diagnosis grid that maps metric to stage to miss to failure is the routing those lists skip. Deleting the grid mentally, the prose still holds: recall diagnoses a miss, precision diagnoses noise, faithfulness diagnoses invention, and none of them diagnose a stale document by themselves.

Low Recall@k or Hit@k is when RAG misses the document: the passage never entered the window, so no generator metric can recover it. Low Precision@k or low contextual precision is why RAG returns the wrong chunk: the window is the wrong neighbours, and a faithful answer from those neighbours will still be the wrong policy. Low context recall can also be when the answer is split across two chunks, because each chunk looks locally relevant and neither contains the full fact. Low faithfulness is hallucination despite context: the list was usable and the model still invented a clause. Answer correctness against a gold string cannot choose among those failures, which is why it sits in the table as a headline rather than as a diagnosis. Online user signals, thumbs and re-asks, live at online evaluation and still do not name the stage. What a high score on any of these rows can still miss is a different list, and it is why a green dashboard is not a healthy system.

What can a high RAG metric score still miss?

A high RAG metric score can still miss anything the metric was not built to see: a stale document, a permission leak, a cost blow-up, an agent that completed the wrong task, and a faithful answer from a passage that is no longer true. Yu et al. (2024) generalise that miss as a property of RAG as a whole. Evaluating retrieval, generation, and the system separately is required, and it is still not sufficient, because the interplay, latency, ambiguous queries, and a changing corpus sit outside any one metric. Their Auepora framework (target, dataset, metric) is the checklist for reading a published number: what pair of evaluable output and ground truth was scored, on which data, with which formula. A faithfulness of 0.9 that does not declare those three is not comparable to a faithfulness of 0.9 that does.

Freshness is the temporal form. Retrieval returns the current index, and the current index can still hold a policy you deleted last week, which is why RAG answers from documents you already deleted. No cell in the diagnosis table has a column for “this chunk’s timestamp.” Cost is the economic form. IBM’s cookbook lists GPU/CPU utilisation (CPU for retrieval, GPU for generation), LLM-call cost, infrastructure cost, and operation cost as their own metric family, and that accounting lives with what a RAG system actually costs to run. Redis’s production section names a related miss: unmonitored LLM spend, redundant similar queries, and cache hit rates that never appear on a faithfulness dashboard. Security is the compliance form. Patronus lists prompt injection and sensitive-data leakage as checks that context and generator effectiveness do not cover, and points at the OWASP list of LLM vulnerabilities as the wider test before production. Those checks are named here so a faithfulness dashboard is not mistaken for a security review.

Agentic extras are the workflow form. Confident AI notes that agentic RAG may need a task-completion metric on top of the five RAG scores, because a pipeline that retrieved the right chunk and wrote a faithful paragraph can still have called the wrong tool. That extra metric is not defined here. The judge that produced the green scores can itself be wrong, which is why Redis still wants periodic human spot checks after you change models or evaluation prompts. Patronus’s remaining best practices, automated testing pipelines, evolving gold references, and drift thresholds, are how a panel stays honest after the first dashboard ships. A metric list that does not say those misses will be quoted as if a high NDCG were a healthy product.

RAG evaluation metrics are a panel that scores retrieval, generation, and context use as separate measurements, because a single accuracy number cannot tell you which stage to fix. Weaviate’s information-retrieval family, Precision@K, Recall@K, MAP@K, MRR@K, and NDCG@K, counts and ranks what entered the window. Faithfulness, answer relevancy, and the context-precision and context-recall judges grade what the model wrote and whether the bundled context was usable. BLEU and ROUGE measure wording overlap with a reference and do not measure grounding. Human raters still own tone, safety, and the sample a dashboard painted green. Read the metric that matches the symptom, and treat a high score as a statement about one relationship, question to chunks, chunks to answer, or answer to gold, never as a statement that the system is true.