RAG Evaluation Metrics: The Complete Catalogue

RAG evaluation metrics are measurements that score retrieval quality, generation quality, or end-to-end context use separately — so you can tell which stage of the pipeline failed. A catalogue without a stage column is a list, not a diagnosis. This page is the catalogue: every metric in common use, what it measures, what it misses, and which failure it can actually point to.

What are RAG evaluation metrics?

RAG evaluation metrics quantify how well a retrieval-augmented generation pipeline retrieves relevant context and generates a grounded answer. They are not a single accuracy number — they are a panel. Every serious guide on the top-ranking results for rag evaluation metrics (GeeksforGeeks, Evidently AI, Agenta, Atlan, GetMaxim, LangCopilot, DeepChecks — captured 2026-07-27) groups that panel into three families:

• Retrieval metrics — did the right chunks come back, and in a useful order?
• Generation metrics — is the answer faithful to what was retrieved and relevant to the question?
• Context / end-to-end metrics — does the retrieved set as a whole contain what the question needs, without drowning in noise?

The split exists because generation metrics grade the answer against retrieved context. They cannot detect a wrong chunk that was answered faithfully. Run the method on how to evaluate a RAG system; use this page to pick which meter to read.

Which metrics measure RAG retrieval?

Retrieval metrics score the ranked list of chunks (or documents) returned for a query against a set of known relevant items. They answer whether the right passage was ever a candidate — before the generator touches it.

• Precision@k — of the top-k results, what fraction are relevant?
• Recall@k — of all known relevant items, what fraction appear in the top-k?
• MRR (Mean Reciprocal Rank) — how high is the first relevant item? Reciprocal of its rank, averaged over queries.
• NDCG (Normalised Discounted Cumulative Gain) — graded relevance with a position discount: a relevant hit at rank 1 counts more than the same hit at rank 10.
• Hit Rate / Hit@k — did any relevant item appear in the top-k? A binary recall floor.
• MAP (Mean Average Precision) — average precision across ranks, then averaged over queries; rewards putting multiple relevant items high.

Formulas and worked scores live on measuring retrieval — this catalogue only names the family.

Which metrics measure RAG generation?

Generation metrics score the written answer. Most need the retrieved context as an input; some also need a reference (gold) answer. Names vary by tool — the jobs do not.

• Faithfulness — are the claims in the answer supported by the retrieved context? (DeepEval, Ragas, and most RAG eval stacks.)
• Groundedness — the same job under a different label; Microsoft Azure AI Foundry and some enterprise stacks prefer this name (Microsoft Learn, RAG evaluators docs).
• Answer relevance / relevancy — does the answer address the question that was asked?
• Answer correctness — does the answer match a reference answer when one exists (reference-based)?
• Hallucination rate — share of claims not supported by context; often the inverse of faithfulness in practice.

Two tools reporting “faithfulness” do not always agree on the score for the same answer — definitions and disagreement live on measuring generation. Ragas-specific assumptions are on Ragas explained.

Which metrics measure context use end to end?

Context precision and context recall sit between retrieval and generation: they score the retrieved set as a whole for the question, often with an LLM judge (Atlan’s four-metric frame; Medium and GetMaxim catalogue the same pair).

• Context precision — of the chunks retrieved, how many are relevant to the question? High noise in the context window lowers this score.
• Context recall — of what is needed to answer, how much appears in the retrieved set? Missing evidence lowers this score even when every retrieved chunk looks “on topic.”

High faithfulness with low context recall means the model stayed faithful to incomplete context — a false sense of safety. Building the labelled or judged sets these metrics need is covered on building a golden test set.

Do BLEU and ROUGE work for RAG evaluation?

BLEU, ROUGE, METEOR and BERTScore measure surface or embedding overlap with a reference answer — they do not measure whether a RAG answer is faithful to retrieved context. GeeksforGeeks and DeepChecks still list them among generation options; treat that as historical carry-over from machine-translation and summarisation eval, not as a RAG diagnosis panel. Use them only when you have a gold reference and care about wording overlap. For groundedness failures, read faithfulness or groundedness instead — generation metrics.

Which RAG metric diagnoses which failure?

Each metric can only see one class of failure. Pick the meter that matches the symptom — a high score on the wrong metric is how teams ship a broken stage. The table maps metric → stage → what a high or low score means → the failure page that owns the fix. None of these offline metrics alone catch a stale index: a pipeline can score 0.95 on faithfulness and still answer from outdated documents (Atlan, 2026) — that case routes to stale index.

When you do not yet know the stage, measure retrieval first, then generation — the order on how to evaluate a RAG system. When the numbers disagree with what users see, add online evaluation.