Measuring Generation: Faithfulness, Groundedness and Answer Relevance

Generation metrics score the written answer — whether its claims are tied to retrieved context (faithfulness / groundedness) and whether it addresses the question (answer relevance). They do not tell you whether the right chunk was fetched. This page shows what each metric computes, and why two tools reporting “faithfulness” can disagree.

What do generation metrics measure?

Generation metrics measure whether the answer is supported by the retrieved context and whether it addresses the question that was asked. They assume retrieval already happened: the judge sees the answer and the context window, not the rest of the corpus. DeepEval’s docs are explicit — a low faithfulness score is a generation problem given that context, not a retrieval problem (DeepEval, Faithfulness FAQ, captured 2026-07-27). Score retrieval first on retrieval metrics; use this page for the generator.

These metrics are usually LLM-as-judge under the hood. Judge model defaults move — DeepEval’s docs listed changing OpenAI default model strings as of this teardown (2026-07-27) — so pin the judge version and re-verify before you treat a score as a gate. Bias depth lives on LLM-as-a-judge.

How does the faithfulness metric work?

Faithfulness is the fraction of claims in the answer that are acceptable given the retrieved context — higher is more consistent with what was retrieved. Ragas documents the steps and a worked score (Ragas docs, Faithfulness, stable concepts):

1. Break the answer into individual statements (claims).
2. Check each claim against the retrieved context.
3. Score = (claims supported by the context) ÷ (total claims).

Their example: question “Where and when was Einstein born?”; context states birth on 14 March 1879; a low faithfulness answer says Germany on 20 March 1879. Statement “born in Germany” is supported; statement “born on 20 March 1879” is not → Faithfulness = 0.5 (Ragas docs).

Faithfulness does not measure world knowledge. A claim that is true in the real world but absent from the retrieved text still fails — DeepEval’s FAQ states that plainly. Low faithfulness is the metric signature of hallucination relative to context.

Is groundedness the same as faithfulness?

Groundedness and faithfulness name the same job — is the answer tied to the provided context without fabrication — but products disagree on the label and on strictness. Microsoft Azure AI Foundry ships Groundedness (LLM-judge alignment with context) and a stricter preview Groundedness Pro that returns a binary result via Azure AI Content Safety (Microsoft Learn, RAG evaluators). deepset (January 2024) publishes groundedness as its anti-hallucination metric for RAG. Ragas and DeepEval document the same intent under the name Faithfulness.

Treat the names as aliases for intent, not as interchangeable scores. A “0.8 groundedness” from one stack is not the same number as a “0.8 faithfulness” from another until you match the predicate and the judge.

How does answer relevance work?

Answer relevance (Ragas) / answer relevancy (DeepEval) scores whether the answer addresses the question — not whether it is factual to the retrieved context. The two major implementations compute that differently:

• DeepEval AnswerRelevancyMetric — extract statements from the answer; score = (relevant statements) ÷ (total statements) against the user input. It never looks at retrieval_context, so a low score points at generator drift, not retrieval (DeepEval, Answer Relevancy docs).
• Ragas answer relevance (classic path) — generate hypothetical questions from the answer and compare them to the original query by embedding similarity (documented in Ragas metric guides; summarised in Sachin Jha’s Ragas vs DeepEval comparison, September 2025).

High relevancy with low faithfulness is the confidently wrong case: on-topic prose whose claims the context does not support (DeepEval Faithfulness FAQ). Pair the two meters; do not read relevancy alone.

Why do two tools disagree on faithfulness?

Both Ragas and DeepEval publish a ratio of “good” claims to total claims. They disagree on what makes a claim good — that predicate difference is why the same answer can pass in one tool and fail in the other.

Independent side-by-side runs (Sachin Jha, September 2025) show the split in practice: Ragas treated “Sam Altman founded OpenAI” as faithful when Altman appeared in a multi-founder context, while DeepEval penalised the exclusivity; on an Einstein year off-by-one, Ragas penalised the mismatch while DeepEval was more forgiving depending on judge configuration. Both agreed when the answer was clearly right or clearly wrong. Do not compare raw faithfulness across tools without aligning the truth test, the judge model, and the prompt template. Pick a stack on evaluation tools compared; Ragas-specific assumptions are on Ragas explained.

What does a low generation score mean?

Read generation meters only after you know retrieval cleared a floor — otherwise you grade a faithful answer to the wrong context.

• Low faithfulness / groundedness — claims go beyond the retrieved context → hallucination.
• High answer relevance + low faithfulness — fluent and on-topic, still unsupported → the same hallucination pattern with a worse UX (users trust it more).
• Low answer relevance — the generator drifted off the question; retrieval may still be fine.
• Generation scores high, business answer still wrong — leave this page: check retrieval metrics and stale index (a pipeline can score 0.95 faithfulness on outdated context — Atlan, 2026).