Ragas Explained: What It Measures and What It Assumes

Ragas is an open-source Python evaluation framework for RAG and LLM apps. It scores a pipeline with LLM-assisted metrics — faithfulness, answer relevancy, context precision and context recall — each on a 0–1 scale. Every number inherits a judge model and a set of assumptions; this page names both, and where the scores disagree with human judgement.

What is Ragas?

Ragas is a Python evaluation toolkit maintained as vibrantlabsai/ragas on GitHub and published on PyPI under the Apache License (ragas 0.4.3 as of 13 January 2026, requiring Python ≥3.9 — PyPI project page). You feed it an evaluation dataset of questions, retrieved contexts, generated answers and — for some metrics — a labelled reference, then call evaluate() to get per-sample and aggregate scores instead of a vibe check.

For RAG the library’s job is to separate retriever failures from generator failures. The four core metrics do that split. The same package also ships agent, SQL and multimodal metrics (listed on the Ragas available-metrics docs); those sit outside this profile. Classic n-gram metrics such as BLEU and ROUGE measure overlap with a reference and ignore retrieved context, which is why RAG-native scores exist — the full catalogue of what to use instead is at RAG evaluation metrics. Orientation across the evaluation cluster is at RAG evaluation.

What do the four core Ragas metrics measure?

Four scores, each aimed at one stage of the pipeline. Higher is better on all four; each is documented to land on a 0–1 scale (answer relevancy notes that cosine similarity is not strictly guaranteed inside that range — Ragas answer-relevancy docs).

• Faithfulness — fraction of claims in the generated answer that can be inferred from the retrieved context. Ragas docs walk a worked case: question “Where and when was Einstein born?”, context gives 14 March 1879, a low answer says 20 March → statement “born in Germany” supported, statement “born on 20 March 1879” not → Faithfulness = 0.5 (Ragas Faithfulness docs, via the published GitHub docs source; docs.ragas.io returned Cloudflare 429 this session).
• Answer relevancy (also called response relevancy) — generate N artificial questions from the answer (default N=3), embed them, and take the mean cosine similarity to the original user query. It scores whether the answer addresses the question, not whether it is factually grounded (Ragas Answer Relevancy docs).
• Context precision — whether relevant chunks sit high in the retrieved ranking: mean of precision@k weighted by a relevance indicator at each rank (Ragas Context Precision docs). Variants exist with a reference, against the response, or without an LLM.
• Context recall — fraction of claims in the labelled reference answer that are attributable to the retrieved contexts. Calculating it always needs a reference (Ragas Context Recall docs).

How faithfulness and relevancy are computed across tools — and why two “faithfulness” scores can disagree — is at measuring generation. Worked information-retrieval formulas for precision and recall sit at measuring retrieval. The matrix below is the Ragas-specific read: each metric beside the assumption that makes the number usable.

Ragas metric capabilities and assumptions, side by side

No metric without the ceiling that rides with it. Each row pairs what Ragas gives you with the assumption that makes the score meaningful — so you know what a high number does not prove.

The two rows that surprise teams first are faithfulness-without-truth and recall-without-labels. Both get their own section rather than a single cell.

What does Ragas assume about your judge model and your labels?

Most Ragas metrics are LLM-assisted: a judge model extracts claims, verifies entailment, or generates reverse questions. The score therefore inherits that judge’s biases and run-to-run variance — the same reason the evaluation-tools survey warns against trusting a single judge run. Practical controls documented by practitioners running Ragas in CI: set judge temperature to 0, pin the judge model version, and re-baseline deliberately when you change either (qaskills Ragas reference, June 2026).

Field requirements split cleanly. Faithfulness and answer relevancy can run without a labelled reference — they compare the answer to retrieved context and to the question. Context recall always needs a reference; reference-based context precision and noise sensitivity use one too (Ragas docs; qaskills FAQ). Building and keeping that labelled set honest is building a golden test set; generating candidates with synthetic pipelines is synthetic evaluation data.

Cost and latency track the judge. Faithfulness is typically the most expensive metric because it fires several LLM calls per sample (claim extraction, then per-claim NLI). Ragas documents an escape hatch for the verification step: FaithfulnesswithHHEM, which can use Vectara’s open HHEM-2.1-Open classifier instead of a full LLM for the entailment check (Ragas Faithfulness docs). Measured judge failure modes — position bias, verbosity bias, self-preference — are catalogued at LLM-as-a-judge.

Where does Ragas disagree with human judgement?

A Ragas score is an LLM judge’s output. It can diverge from a human rater, and from another evaluation library, on the same sample — that is not a bug in the spreadsheet, it is the metric’s design.

Two disagreement surfaces are documented without inventing scores:

• Against another tool’s predicate. Ragas faithfulness asks whether each claim can be inferred / supported from the retrieved context. DeepEval’s published faithfulness metric treats a claim as truthful if it does not contradict the retrieval context — a weaker test on the same formula shape. Independent side-by-side runs (Sachin Jha, September 2025) show the split: on “Sam Altman founded OpenAI” with a multi-founder context, Ragas accepted the answer while DeepEval penalised the implied exclusivity; on an Einstein year off-by-one, Ragas penalised the mismatch while DeepEval was more forgiving. The full predicate table is at measuring generation.
• Against a human who knows the corpus or the world. High faithfulness on the wrong retrieved chunks looks good to the metric and bad to a human — the answer is faithful to garbage. A true fact that never appeared in the retrieved context fails faithfulness even when a world-knowledge human would accept it; Ragas is scoring consistency with what was retrieved, not encyclopedic truth (Ragas Faithfulness definition; same pattern in DeepEval’s FAQ as cited on the generation-metrics page).

Neither surface means “do not use Ragas”. It means read the score as what it claims — judge-mediated consistency with the inputs you passed — and calibrate against humans on a slice of your own domain. The scored pick among Ragas, DeepEval, TruLens, Phoenix and Braintrust is at RAG evaluation tools compared.

Is Ragas open source, and how does it compare to DeepEval?

Yes. The library is Apache-licensed and free to install (pip install ragas); you pay for the judge LLM and embeddings you point it at, not for the framework itself (PyPI, January 2026). Against the tool it is most often weighed against:

• DeepEval — also open-source; assertion- and metric-library oriented, with statement-level answer relevancy and a non-contradiction faithfulness predicate rather than Ragas’s reverse-question relevancy and strict entailment check (Jha, 2025; generation-metrics sibling).

Which of the two wins for a given pipeline is not this profile’s call — crowning either would be the opposite of the neutrality the tools survey is built for. The scored five-tool verdict is at RAG evaluation tools compared.