How to Evaluate a RAG System

Evaluating a retrieval-augmented generation system means scoring retrieval and generation as separate measurements, because a single end-to-end score cannot tell you which stage to fix. Yu, Gan, Zhang, Tong, Liu and Liu’s survey “Evaluation of Retrieval-Augmented Generation” (arXiv 2405.07437v2, 3 July 2024) states the reason: the hybrid structure and dynamic knowledge sources make component scores and whole-system scores different jobs. Evidently AI’s RAG evaluation guide (captured 17 September 2026) puts the debugging question in practitioner terms: when an answer is wrong, was the retriever unable to find the right documents, or did the model hallucinate even though the context was correct? The method is therefore three layers, retrieval quality, generation quality, and only then end-to-end and production checks, using labelled queries, LLM judges, or both. Vector databases such as Weaviate, Qdrant, and Pinecone change what retrieval can return; they do not replace the measurement.

Why does RAG evaluation matter?

RAG evaluation matters because a pipeline that looks fine end to end can still be broken at one stage, and without stage-level scores the fix lands on the wrong one. Comet’s RAG evaluation guide (captured 17 September 2026) states the diagnostic problem directly: retrieval misses, generator hallucinations, and answers to a different question than the one asked all look identical from the outside, a wrong answer, and each needs a different repair. Qdrant’s evaluation guide names the same three stages, retrieve, augment, generate, as places errors occur, and treats evaluation as the way to keep a RAG application useful after the first demo.

Evidently AI lists four moments in the product lifecycle that all need measurement, with different datasets: development experiments on synthetic or curated offline sets; stress and adversarial testing on edge cases before launch; production monitoring on live queries; and regression testing on a frozen set of known queries after every change. Google Cloud’s RAG retrieval post adds a human-evaluation step after automated metrics, because some failure modes, tone, completeness, and safety, still need a rater. Skipping that stack is how a team ships a chatbot that quotes the handbook faithfully and still answers the wrong policy.

Yu et al. (2024) split the evaluation challenge the same way the pipeline is split. Retrieval is hard because the knowledge base is large, mixed in quality, and changing in time, so precision, recall, and relevance have to be read against a moving corpus. Generation is hard because faithfulness, accuracy, and relevance are partly subjective on open-ended tasks, and a “correct” answer is not a single string. The whole system is hard because the two components interact: a retrieval gain that the generator ignores is not an end-to-end gain, and a generator that writes well from the wrong passage will look healthy to an answer-only score. Those three paragraphs of the survey are the reason a blended quality number is not a diagnosis.

Google Cloud’s optimizing-RAG-retrieval post (captured 17 September 2026) sequences the same work as three steps: create a testing framework with example metrics, run root-cause experiments on retrieval variables, then add human evaluation. Common frameworks it names include the usual open evaluators; the post’s point is the order, framework, then diagnosis, then raters, not a ranked list of products. That order is compatible with Evidently’s lifecycle and with Weaviate’s bubble-up rule: automated retrieval scores first, generation scores second, humans on a sample before a launch that quotes policy.

Weaviate’s “An Overview on RAG Evaluation” (the Weaviate blog post inspired by a conversation with the Ragas authors) splits the pipeline into indexing, retrieval, and generation, and notes that errors bubble up: an indexing fault becomes a retrieval miss, which becomes a generation that looks fluent. Approximate nearest-neighbour recall, whether the index returns the true nearest vectors, is a different number from labelled document relevance, and mixing the two hides the broken layer. Once evaluation is treated as stage diagnosis rather than a single quality grade, the next question is the method: what to score first, and in what order.

How do you evaluate a RAG system?

You evaluate a RAG system by scoring retrieval quality and generation quality as separate measurements, then checking end-to-end behaviour only after both stages clear a floor. Evidently AI’s guide says it is often more useful to evaluate retrieval and generation separately than to score only the final answer, because teams iterate on those components independently and because a bad answer has two different causes. Yu et al. (2024) call the same split a challenge of the whole system: performance cannot be fully understood by evaluating each component in isolation, and yet isolated scores are what tell you which component to change.

The three layers, with the family each one owns, are retrieval, generation, and then the system. Retrieval asks whether the right chunks came back, and in a useful order: Precision@k, Recall@k, MRR, and NDCG, worked on real numbers at retrieval metrics. Generation asks whether the answer is faithful to what was retrieved and relevant to the question: faithfulness, groundedness, and answer relevance, defined at generation metrics. End-to-end and production add context precision and recall, task completion, latency, and cost, plus the online signals that a frozen test set cannot see. A high generation score on bad retrieval is a false pass: the model answered faithfully from the wrong context.

LangSmith’s evaluate-RAG tutorial (LangChain docs, captured 17 September 2026) implements that split as four evaluators on one dataset: correctness of the response against a reference answer, relevance of the response to the input, groundedness of the response against retrieved docs, and relevance of retrieved docs to the input. That is the same three-layer method with the end-to-end check (correctness versus a gold answer) kept distinct from the two stage checks. The catalogue of every metric in common use, what each measures, what it misses, and which stage it can diagnose, lives at RAG evaluation metrics. The layer that most teams under-build first is retrieval, because it is the older information-retrieval problem sitting under a new generator.

How do you measure retrieval quality in RAG?

Retrieval quality is measured by comparing what the retriever returned with what should have been returned, at a cutoff k that matches the number of chunks the prompt actually reads. Evidently AI lists the standard information-retrieval metrics for a ground-truth set: Precision@k (of the top k items, how many are relevant), Recall@k (of all relevant items, how many appeared in the top k), Hit Rate (whether at least one relevant item appeared), and NDCG@k (a rank-aware score that rewards relevant items appearing higher). Qdrant lists Precision@k, MRR (position of the first relevant document), and DCG/NDCG as the semantic-retrieval metrics to watch while tuning an index.

Weaviate’s retrieval-evaluation-metrics article walks the same family, Precision@K, Recall@K, MAP@K, MRR@K, and NDCG@K, and notes that NDCG is the one that needs graded relevance (relevant versus somewhat relevant versus not) rather than a binary label. Binary labels are cheaper and more common. Graded labels are what NDCG is for. None of those metrics can be computed without knowing which documents are relevant for each query, which is why a labelled set is not optional if recall is the number you care about. Evidently AI is explicit about the cheaper alternative: label the retrieved results after the fact as relevant, partially relevant, or not. That method cannot compute recall, because it never defined what should have been retrieved.

A third path, LLM-judged relevance, asks a model whether a chunk is useful for a query. Weaviate’s RAG-evaluation post gives the prompt shape: “How many of the following search results are relevant to the query {query}? {search_results}” as a proxy for precision, and “Do these search results contain all the needed information to answer the query?” as a proxy for recall. Those proxies are cheaper than full labelling and weaker than it. Rank order still matters even when a relevant chunk is present. Liu et al., “Lost in the Middle” (Transactions of the Association for Computational Linguistics, 2024; arXiv 2307.03172), document a U-shaped attention curve that buries mid-prompt evidence, a finding applied to RAG in Comet’s guide and, separately, in Qdrant’s. A retriever that returns the right chunk at position 8 of 10 can still lose the answer. The position failure in full is lost in the middle. Retrieval scores tell you whether the right text arrived. They do not tell you whether the generator used it, which is the next measurement.

How do you measure generation quality in RAG?

Generation quality is measured against the retrieved context and against the question, not against the model’s general knowledge. Weaviate’s RAG-evaluation post, summarising Ragas, names two generation scores: faithfulness, how factually correct the answer is based on the retrieved context, and answer relevancy, how relevant the answer is given the question. An answer can score high on faithfulness and low on relevancy: copying the context verbatim is faithful and not an answer. The same Weaviate post cites LlamaIndex’s FaithfulnessEvaluator, which scores whether the response matches the retrieved context, groundedness under another name.

LangSmith’s tutorial grades three generation-side properties with an LLM: correctness versus a reference answer, relevance versus the user input, and groundedness versus the retrieved documents. Patronus groups generator effectiveness separately from context effectiveness, and still lists BLEU and ROUGE as older overlap metrics that do not capture meaning. Comet’s guide is blunt about why those overlap metrics lost the job: a response that conveys the correct information in different words scores poorly under BLEU, while an LLM judge can recognise the semantic equivalence. That shift is why most current RAG scores are themselves model calls, with the costs and biases that implies.

Two tools reporting “faithfulness” do not automatically agree, because the prompt, the judge model, and the definition of a supported claim differ. That disagreement, and the prompts that produce it, belong with the generation-metric definitions. The practical rule is smaller: never let a generation score stand in for retrieval. A faithful answer from the wrong policy document is a retrieval incident that a faithfulness dashboard will paint green. Comet’s parental-leave example is the worked case: “Employees are eligible for 12 weeks of parental leave” can be faithful to a benefits chunk and still miss the 12-month tenure rule that lived in a different document. Faithfulness asked the wrong question. The triad is how those questions get separated on purpose.

What is the RAG triad?

The RAG triad is three relationships scored on every interaction: relevance of retrieved context to the query, faithfulness of the generated answer to that context, and relevance of the answer to the original question. Comet’s guide attributes the name to TruLens and notes that DeepEval adopted the same framing. Think of it as three questions: did the retriever find relevant information, did the generator stick to the facts it was given, and did the final answer address what the user asked?

The triad is a diagnostic, not a single score. Comet’s HR assistant example shows why collapsing it hides the fix. A user asks whether they are eligible for parental leave after 11 months. The system answers with the 12-week benefit and skips the tenure minimum. If the eligibility document was never retrieved, the first leg failed and chunking or search is the repair. If the eligibility document was retrieved and ignored, the second or third leg failed and the prompt or the model is the repair. End-to-end accuracy would have marked both runs as wrong and left the team guessing.

Ragas, as described in Weaviate’s evaluation post, builds a composite from four zero-shot LLM prompts that map onto the same triangle: Faithfulness and Answer Relevancy on the generation side, Context Precision and Context Recall on the retrieval side. That composite is useful as a dashboard headline and dangerous as the only number, because a rise in the headline can be a rise in one leg and a fall in another. The profile of those assumptions, and where they disagree with human raters, is Ragas explained. The triad also assumes you have something to score against, and assembling that labelled set is the expensive part.

What do you need in a RAG evaluation dataset?

A RAG evaluation dataset is a set of questions paired with the evidence that would make an answer right: relevant document or chunk IDs for retrieval, and, when generation is scored against a reference, a gold answer. Evidently AI calls this a custom retrieval benchmark, an exam for the system. LangSmith’s tutorial builds the same object as a dataset of examples with inputs and reference outputs, then runs evaluators over it. Without that pairing, Precision@k and Recall@k cannot be computed, and generation “correctness” has nothing to be correct against.

Building the set by hand is slow and goes stale when chunk boundaries move, which Evidently AI flags as the catch: change the chunker and every chunk-ID label has to be revisited. Hugging Face’s RAG-evaluation cookbook (Open-Source AI Cookbook, captured 17 September 2026) answers that cost with a synthetic pipeline: source documents, question-generation agents, and critique agents that filter the generated questions before they enter the set. The cookbook then builds a retriever and a reader on the same knowledge base and benchmarks the pair, so the set and the system are measured together rather than invented separately. Synthetic questions scale. They also inherit the generator’s vocabulary and the documents they were born from, so a set built only from easy paragraphs will not catch the queries users actually type. The bias, the critique-agent checks, and the cases where synthetic data is usable live at synthetic evaluation data.

How many questions is enough is a property of the decision you want to make, not a universal n. A change to an embedding model needs enough queries that a recall movement is not noise. A production regression suite needs the queries that have already failed, plus a sample of current traffic. Public leaderboards do not substitute for that set: Yu et al. (2024) spend a section of Auepora on how existing RAG benchmarks differ in target, dataset, and metric, and why a rank on someone else’s corpus rarely transfers. The transfer limit, and what MIRAGE, RAGBench, and the rest actually test, is public RAG benchmarks. The how-to for size, source, and honesty rules is building a RAG test set. Most of those scores, on a set you do hold, will be produced by another language model acting as a judge, which is a measurement instrument with its own failure modes.

How do LLM judges score a RAG system?

LLM-as-a-judge scores a RAG system by prompting a language model to grade retrieved chunks or generated answers against a rubric, then treating that grade as the metric. Weaviate’s evaluation post describes the spread of zero-shot LLM evaluation, a prompt template with no labelled examples, as the technique Ragas pioneered for RAG, and notes three knobs: the metric design, the exact language of the prompt, and the model used as judge. Few-shot evaluation adds labelled examples to the prompt (in-context learning). ARES (Saad-Falcon et al.), summarised in the same Weaviate post, fine-tunes lightweight classifiers for context relevance, answer faithfulness, and answer relevance from synthetic queries plus a small human validation set, and reports that those fine-tuned judges outperformed zero-shot and few-shot GPT-3.5-turbo-16k in the paper’s own split.

Comet’s guide names two further judge recipes that have become common infrastructure. G-Eval (Liu et al., EMNLP 2023) has the judge generate step-by-step criteria before scoring and uses token probabilities for a continuous score rather than an integer. Prometheus (Kim et al., ICLR 2024) showed that open-source models fine-tuned for evaluation can match GPT-4’s correlation with human judges when given structured rubrics, which is the reason a team is not locked to a proprietary judge. Patronus still lists embeddings-based comparison alongside LLM judges, because a cosine between answer and reference is cheaper than a second model call and weaker at catching factual insertions.

Judges fail in patterned ways. Position bias, verbosity bias, and self-preference are the named ones, and they are the reason an LLM grade is not a human grade. Weaviate’s cost napkin in that post is a reminder that the instrument is also a bill: their illustrative GPT-4 arithmetic for 100 queries at roughly 6,000 tokens each, at a stated $0.005 per 1K tokens, comes to $3 for zero-shot and five times that for a 5-example few-shot prompt. Those dollar figures are the blog’s own toy example, not a current price list. The failure-mode depth, agreement rates with human raters, and how to set a judge up without fooling yourself, sit at LLM-as-a-judge. Once the instrument is chosen, the remaining product question is which packaged tool runs it, and that question has no single winner, only stages.

Which RAG evaluation tools measure which stage?

RAG evaluation tools are packages that run retrieval metrics, generation metrics, or traces, and the right first tool is the one that scores the stage you cannot see yet. Qdrant’s guide recommends Ragas for question-answer-context datasets (faithfulness, relevance, semantic similarity) and Arize Phoenix for step-by-step traces that show where a response was built, plus latency, token usage, and errors. Patronus surveys Ragas, DeepEval, and TruLens as the common open evaluators, then its own platform. LangSmith is the tracing-and-dataset product LangChain’s tutorial uses to store examples and run the four evaluators above. Comet’s Opik is the product that guide uses to implement ContextPrecision, ContextRecall, and G-Eval.

None of those tools is a vector database, and a vector database is not an evaluation suite. Weaviate publishes the metric definitions and the Ragas conversation that informed its blog posts; it does not replace Ragas, DeepEval, TruLens, Phoenix, or LangSmith as the thing that scores a run. Qdrant publishes an evaluation guide because retrieval quality is a property of the index; the guide still points at Ragas and Phoenix for the scores. A vendor blog that grades its own product is not a bake-off. The scored, tool-neutral comparison, Ragas, DeepEval, TruLens, Phoenix, and Braintrust on the same pipeline, is RAG evaluation tools.

A tool that only grades the final answer will hide the retrieval and generation split. A tool that only traces spans will not compute Recall@k unless you attach labels. Start with retrieval labels if you have them; start with a judge on context relevance if you do not; add a generation judge after retrieval clears a floor. Hugging Face’s cookbook is the open recipe for people who want to build the set and the RAG system in one notebook rather than buy a platform. It generates questions from source documents, filters them with critique agents, then benchmarks a retriever and a reader on that same knowledge base, so the exam and the system are measured together. The platform choice is secondary to keeping the three layers distinct once the system is in production, where the dataset starts to include real users.

How do you keep evaluating RAG after it ships?

You keep evaluating RAG after it ships by attaching scores to live traces, freezing a regression set that must not get worse, and sampling production traffic into the next test set. Evidently AI’s lifecycle table is the map: production monitoring uses live user queries, often session-level, and watches faithfulness, completeness, safety, and refusals; regression testing uses curated offline datasets of happy paths and edge cases to catch silent failures after updates. Qdrant points at Phoenix for the visual deconstruction of a single response, which is the tracing half of that loop. Span-level tracing from query to answer, and the traces that identify almost every failure, is RAG tracing and observability.

Regression in CI is a different job from a dashboard. DeepEval positions itself as unit-testing RAG in a build. Qdrant’s guide treats that CI shape as the regression job; Patronus surveys the same open evaluators. Non-determinism is the catch: the same prompt can yield a different generation, which Weaviate’s RAG starter guide also states as expected model variability. A CI assertion therefore has to tolerate that variance, by judging properties (grounded, relevant, citation present) rather than exact strings, or by averaging a small number of samples. What to assert, what to sample, and what to do about non-determinism is RAG regression testing.

Online evaluation turns production into the set. Implicit signals, thumbs, copy events, immediate re-asks, and escalation to a human, are weaker than a labelled chunk ID and cheaper to collect at volume. Sampling rates decide whether you can see a drift in a week or only in a quarter. Evidently AI also lists stress and adversarial testing as a pre-production stage of their own: edge cases, red-teamed prompts, prompt-injection resistance, hallucination checks. Those attacks are a security concern as well as an evaluation concern; the user-signal half of the production loop is online evaluation. Even a green dashboard on all of those loops can still be wrong in a way stage metrics were never designed to see.

Can a RAG system score well and still give wrong answers?

A RAG system can score well and still give wrong answers, because most RAG metrics grade the relationship between question, retrieved context, and answer, not the relationship between that context and the world. Evidently AI’s debugging fork is the first form of the miss: a faithful answer to the wrong documents. Comet’s parental-leave run is the same miss with a named document split. Weaviate’s bubble-up note is the indexing form: ANN recall can look healthy while labelled relevance is poor, because the neighbours of the query vector are not the documents a rater would mark relevant. Freshness is the temporal form: retrieval returns the current index, and the current index can still hold a policy you deleted last week.

Yu et al. (2024) generalise that miss as a property of RAG as a whole. Evaluating retrieval, generation, and the system separately is required, and it is still not sufficient, because the interplay, latency, ambiguous queries, and a changing corpus sit outside any one metric. Their Auepora framework (target, dataset, metric) is the checklist for reading a published benchmark: what pair of evaluable output and ground truth was scored, on which data, with which formula. A number that does not declare those three is not comparable to a number that does. Public leaderboard ranks fail that test often enough that they belong on RAG benchmarks, not in a launch email.

The practical response is to keep the stages apart, then add the checks the metrics cannot see: index presence, document timestamps, permission tests, and a human pass on a sample of production answers. When a symptom is already in hand, the taxonomy of how RAG systems fail names the stage. When only “something is wrong” is in hand, how to find which stage of your RAG pipeline broke is the diagnostic order. Evaluation is how you know the stage. The failure taxonomy is what to do once you know.

Retrieval-augmented generation is a retrieve-then-generate pipeline, and evaluating it is a retrieve-then-generate measurement: score whether the right chunks came back, score whether the answer stayed inside those chunks and addressed the question, and only then trust an end-to-end number. Lewis et al. (2020) evaluated knowledge-intensive NLP tasks with a jointly trained retriever and generator. Production stacks evaluate a frozen model over a private index with labelled queries, LLM judges, and live traces, which is a different experiment that still fails in the same two places. Formulas, datasets, judges, packaged tools, and the production loop each have a dedicated page under evaluation. The order those measurements run in is retrieval first, generation second, production third, so a green score cannot hide a broken stage.