Measuring Generation: Faithfulness, Groundedness and Answer Relevance

Generation metrics score the written answer against retrieved context and against the question, so you can tell whether the generator stayed inside the window and whether it answered what was asked. They do not score whether the right chunk was fetched. Weaviate’s overview of RAG evaluation (Erika Shorten and Connor Shorten, 21 November 2023) treats generation as the top of the stack and names two Ragas scores for it: Faithfulness, how factually correct the answer is based on retrieved context, and Answer Relevancy, how relevant the answer is given the question. DeepEval’s RAG triad (docs captured 18 September 2026) splits that generator job into faithfulness and answer relevancy, and keeps contextual relevancy for the retriever. Openlayer’s production guide (13 July 2026) adds a third generator label, groundedness, and treats it as a distinct failure from faithfulness. How those scores become a launch decision is the complete method, tool-neutral.

What do generation metrics measure?

Generation metrics measure what the language model does with the context it was given, not whether that context was the right context. They assume retrieval already happened: the judge sees the answer and the retrieved window, not the rest of the corpus. Weaviate is explicit that generation evaluation commonly assumes oracle context, or controlled distractors such as the Lost in the Middle experiments, when scoring faithfulness and answer relevancy. A high generation score on a wrong window is still a wrong product answer. Score the ranked list first on retrieval metrics; use these meters for the generator.

DeepEval’s RAG triad names three scores and maps each to a hyperparameter. Answer relevancy grades the generated answer against the user input and, in DeepEval’s wording, mainly diagnoses the prompt template. Faithfulness grades the answer against retrieval context and diagnoses the generator model. Contextual relevancy grades the retrieved chunks against the input and diagnoses chunk size, top-k, and the embedding model. That third meter is a retrieval score; the family list that holds both planes sits with RAG evaluation metrics. Openlayer’s July 2026 production post splits the same pipeline into three failure surfaces that a single accuracy number collapses: retrieval quality, faithfulness, and groundedness. A response can look fine in a demo and still contradict the source sitting in context, invent a statistic that never appeared, or answer a different question than the one asked.

These scores are usually LLM-as-judge under the hood. Weaviate’s Ragas conversation frames the RAGAS score as four zero-shot prompts, two for generation and two for retrieval. DeepEval’s faithfulness and answer-relevancy docs (captured 18 September 2026) show example calls with model=”gpt-4.1″ and a default threshold of 0.5, with worked examples using 0.7; the default model string in the parameter list is unpublished in that snapshot, so pin the judge you actually run. Judge defaults move. Bias, agreement, and why two judges disagree on the same pair live on LLM-as-a-judge. What the faithfulness fraction actually counts is the next question, because a named metric without a predicate is still a slogan.

How does the faithfulness metric work?

Faithfulness is the fraction of claims in the answer that are acceptable given the retrieved context, with higher scores meaning more consistency with what was retrieved. Ragas documents the steps and a worked score in its Faithfulness concept page (docs.ragas.io, stable, re-fetched 18 September 2026): identify the claims in the response, check each claim against the retrieved context, then divide supported claims by total claims. A response is faithful, in Ragas’s wording, if all its claims can be supported by the retrieved context. The Super Bowl toy in that same page scores 1.0 when the answer dates the first game to 15 January 1967 and the context states that the First AFL-NFL World Championship Game was played on that date at the Los Angeles Memorial Coliseum.

Ragas’s Einstein example is the published 0.5. Question: where and when was Einstein born. Context: Albert Einstein (born 14 March 1879) was a German-born theoretical physicist. High-faithfulness answer: Einstein was born in Germany on 14th March 1879. Low-faithfulness answer: Einstein was born in Germany on 20th March 1879. Step 1 splits the low answer into “Einstein was born in Germany” and “Einstein was born on 20th March 1879.” Step 2 marks the first as inferable from the context and the second as not. Step 3 is 1 / 2 = 0.5. That 0.5 is Ragas’s published number, not a recompute. Faithfulness here is not world knowledge. The date 20 March 1879 is simply not supported by the retrieved sentence.

DeepEval’s FaithfulnessMetric uses the same claim-extraction shape and a different truth test. It extracts claims from actual_output, then classifies whether each claim is truthful based on facts in retrieval_context. A claim is truthful, in DeepEval’s wording, if it does not contradict any facts presented in that context. Optional flags change the edges: penalize_ambiguous_claims (default false) stops counting ambiguous claims as faithful, and truths_extraction_limit caps how many facts from context the judge may consider. DeepEval’s FAQ states that a real-world truth absent from the retrieved text still counts as unfaithful, and that a low score is a generation problem given that context, not a retrieval problem. Weaviate also names LlamaIndex’s FaithfulnessEvaluator as a score based on whether the response matches the retrieved context. The pipeline attempt that tries to keep answers inside evidence is grounding. The failure this meter is built to catch is hallucination relative to context. Whether other products call that same job groundedness is the naming fight that follows.

Is groundedness the same as faithfulness?

Groundedness and faithfulness name the job of tying the answer to provided context, but products do not agree on whether those names are synonyms. Openlayer’s July 2026 guide says many RAG evaluation guides treat them as synonyms and that they are not. Groundedness, in Openlayer’s split, measures whether a response is supported by retrieved context: a response that invents a statistic not present in any retrieved chunk fails groundedness even if the rest of the answer reads correctly. Faithfulness, in that same split, measures whether the response accurately represents the source without distorting, omitting, or inverting its meaning: citing 10 percent when the source says 10x fails faithfulness even though the chunk was in context. Openlayer’s examples of mixed failure are a grounded but unfaithful inversion of a correlation sign, and a faithful citation of a source that was never retrieved for this query.

Braintrust’s RAG-evaluation article (captured 18 September 2026) reports the opposite working assumption. Answer groundedness measures whether every claim can be traced back to retrieved context. Faithfulness and groundedness are often used interchangeably in the literature, with a subtle difference: faithfulness refers to overall truthfulness relative to sources, while groundedness focuses on verifying individual claims at a more granular level. In practice, Braintrust says both metrics detect the same core failure, claims not supported by retrieved context. Deepchecks’s metrics post (captured 18 September 2026) sits closer to the claim-level reading: faithfulness is operationalised as natural-language-inference pipelines that classify atomic claims as entailed, neutral, or contradicted, citing DeBERTa-style models, while groundedness emphasises traceability through attribution maps that link each claim to a source span. Thresholds above 0.9 indicating that outputs are “grounded” are Deepchecks’s wording for that NLI aggregation, not a published RAG launch bar.

Microsoft’s path-to-a-golden-dataset post (Alexandra Savelieva and Sandip Kulkarni, Data Science + AI at Microsoft, 21 May 2024) documents Azure AI Studio’s Groundedness as how well answers align with information from the source data: even if the responses are factually correct, they are considered ungrounded if they cannot be verified against the provided sources. That is the same “true in the world, absent from context” rule DeepEval states for faithfulness. Weaviate’s 2023 overview does not use the groundedness label for generation; it uses Faithfulness and Answer Relevancy from Ragas. Treat the names as aliases for intent, not as interchangeable scores. A 0.8 groundedness from one stack is not the same number as a 0.8 faithfulness from another until you match the predicate and the judge. Openlayer publishes starting thresholds “reported by practitioners”: flag groundedness below 0.80 for review, block faithfulness below 0.70 before the user sees it. Those two numbers are Openlayer’s stated starting points, adjustable per corpus, not a site-wide gate. Answer relevance is the other generator meter, and it does not look at context at all in DeepEval’s implementation.

How does answer relevance work?

Answer relevance scores whether the answer addresses the question, not whether it is factual to the retrieved context. Weaviate describes Ragas Answer Relevancy as how relevant the answer is given the question, and notes that an answer can have high faithfulness and low answer relevance: a faithful response that copies the context verbatim would score low on relevance. The same Weaviate paragraph says the relevance score is penalised when an answer lacks completeness or has duplicate information. That is a usefulness meter sitting next to a support meter, not a second name for faithfulness.

DeepEval’s AnswerRelevancyMetric (docs captured 18 September 2026) extracts statements from actual_output, classifies whether each statement is relevant to input, and scores relevant statements over total statements. Required arguments are input and actual_output only. It never looks at retrieval_context, so a low score points at generator rambling, hedging, or drift, not at retrieval. DeepEval’s FAQ is explicit that the metric does not check whether statements are grounded, that an on-topic answer can still hallucinate, and that preamble, caveats, or tangents drag the score down even when the core answer is correct. Default threshold in that snapshot is 0.5; worked examples use 0.7. No reference answer is required, which is why the metric can run on live traffic.

Ragas’s classic answer-relevance path, as summarised by Sachin Jha (Data Scientist II at Red Hat, “Ragas vs DeepEval,” 26 September 2025), generates hypothetical questions from the response and compares those questions to the original query by embedding similarity. That is a different computation from DeepEval’s statement-level judge, even when both outputs are labelled 0 to 1. High relevancy with low faithfulness is the confidently wrong case: an on-topic answer whose claims the context does not support. DeepEval’s Faithfulness FAQ names that pattern and attributes it to the model leaning on parametric knowledge instead of retrieved evidence. Weaviate’s verbatim-copy example is the opposite corner: high faithfulness, low relevance. Pair the two meters. Do not read relevancy alone, and do not treat a Ragas relevance number as a DeepEval relevancy number. Why two faithfulness implementations still disagree, even when both claim a claims-over-total ratio, is the comparison that decides whether two stacks can share a number.

Why do two tools disagree on faithfulness?

Ragas and DeepEval both publish a ratio of accepted claims to total claims. They disagree on what makes a claim acceptable, and that predicate difference is why the same answer can pass in one tool and fail in the other. Ragas asks whether each claim can be inferred from, or supported by, the retrieved context. DeepEval asks whether each claim does not contradict facts in retrieval_context. Inferable is a tighter test than not-contradictory: a plausible extra detail can survive a contradiction check and still fail entailment.

The table is the documentation split. Independent side-by-side runs show it in answers. Sachin Jha (26 September 2025) scored four cases with Ragas Faithfulness plus ResponseRelevancy and DeepEval FaithfulnessMetric plus AnswerRelevancyMetric, both wrapped around prometheus-eval/prometheus-8x7b-v2.0 at temperature 0.2, not DeepEval’s gpt-4.1 example default. On “Who founded OpenAI?” with a multi-founder context, the answer “Sam Altman founded OpenAI” scored higher under Ragas, which treated Altman’s presence as support, while DeepEval penalised the exclusivity implied by naming one founder. On Einstein’s relativity year, context said 1915 and the answer said 1916: Ragas penalised the mismatch as a contradiction against 1915, while DeepEval accepted 1916 as close enough depending on judge configuration. Both agreed high on Vivaldi composing The Four Seasons. Both agreed low on “Python 3.11 was released in July 2021” against an October 2022 context. Jha’s takeaway is that Ragas emphasises logical entailment and DeepEval emphasises pragmatic interpretation; those are Jha’s readings of his four-case run, not primary-docs claims.

Do not compare raw faithfulness across tools without aligning the truth test, the judge model, and the prompt template. Jha’s numbers are a prometheus-8x7b-v2.0 run. DeepEval’s example snippets use gpt-4.1. Ragas’s Einstein 0.5 is a documented walk-through, not a live judge call. Pick a stack on evaluation tools compared. Ragas-specific assumptions sit on Ragas explained. A low score still has to be read as a failure class, or the metric becomes a vanity number.

What does a low generation score mean?

A low generation score points at the generator only after retrieval has cleared a floor. Weaviate’s bubbling-errors note is the reason: indexing and retrieval mistakes show up in generation, but a generation mistake does not rewrite the index. DeepEval’s Faithfulness FAQ is the operational version: the metric grades actual_output against already-retrieved retrieval_context, so a low score means unsupported claims given that window. If that window was the wrong window, you are scoring a faithful summary of the wrong policy. Openlayer’s cascade is the same warning from the other direction: retrieval quality sits upstream of every faithfulness and groundedness score, and a perfectly calibrated generator will still write a plausible answer on irrelevant evidence.

Low faithfulness or groundedness means claims went beyond, or misrepresented, the retrieved context. That is the metric signature of hallucination relative to context, including world-true claims the window does not contain. High answer relevance with low faithfulness is the confidently wrong case DeepEval names: on-topic prose users are more likely to trust, still unsupported. Low answer relevance with high faithfulness is Weaviate’s verbatim-copy case: the model stayed inside the window and still failed to answer the question, often by dumping extra context or missing the asked fact. Low relevance with low faithfulness is drift plus invention; fix the prompt and the model, not the retriever first.

Generation scores that look high while the business answer is still wrong are no longer a generator diagnosis. Check information-retrieval metrics for a missed or buried passage, then stale index for a window that is faithful to an outdated chunk. Openlayer’s 0.80 groundedness review flag and 0.70 faithfulness block are starting thresholds that vendor published for its own gates, not numbers that transfer across corpora. DeepEval’s default 0.5 pass line is a library default, not a published RAG launch bar. Pin the judge, name the predicate, and read the pair of meters together: support against context, usefulness against the question.

Faithfulness, groundedness, and answer relevance are three labels on two jobs: did the answer stay inside the retrieved window, and did it address the question. Weaviate’s Ragas pairing still holds as the clean split. Openlayer’s distinct-failure reading and Braintrust’s interchangeable-in-practice reading can both be true of the products they describe, which is why a 0.8 from one stack is not a 0.8 from another. Compute the claims ratio against a named predicate, pair it with relevance, and only then treat the number as a generator diagnosis.