Ragas Explained: What It Measures and What It Assumes

Ragas is an open-source Python evaluation framework that scores RAG pipelines with LLM-assisted metrics for faithfulness, answer relevancy, context precision, and context recall. The library ships as vibrantlabsai/ragas on GitHub and installs with pip install ragas (GitHub README and ragas.io, captured 18 September 2026). Es, James, Espinosa-Anke, and Schockaert introduce it in arXiv 2309.15217 as a reference-free suite for retrieval-augmented generation assessment. Every number inherits a judge model and a set of assumptions. How those scores become a launch decision is how to evaluate a RAG system.

What is Ragas as an open-source RAG evaluation framework?

Ragas is a Python evaluation toolkit for LLM applications with a RAG-native metric suite. The GitHub README (vibrantlabsai/ragas, captured 18 September 2026) frames it as objective metrics, automatic test-data generation, and feedback loops for LLM apps, with integrations named for LangChain and major observability tools. Atlan’s April 2026 comparison (Emily Winks) expands the brand as Retrieval Augmented Generation Assessment and states that Ragas focuses on four RAG-specific metrics with a lightweight Python setup. The same Atlan article lists the project as open source, notes a commercial Ragas Cloud tier beside the free library, and attributes usage to engineering teams at AWS, Microsoft, Databricks, and Moody’s.

The design problem the paper names is that RAG evaluation has several dimensions at once: whether retrieval returned focused passages, whether the generator stayed faithful to those passages, and whether the generation itself answered the question. Es et al. (arXiv 2309.15217, revised 28 April 2025) argue that a suite of metrics can cover those dimensions without forcing every run to wait on ground-truth human annotations. That reference-free pitch is why teams reach for Ragas early. It is also why some scores still need a labelled reference later: context recall cannot invent the evidence that should have been retrieved.

Weaviate’s overview of RAG evaluation (Erika Shorten and Connor Shorten, 21 November 2023), written after a conversation with the Ragas authors, maps the same four names onto the pipeline: Faithfulness and Answer Relevancy on the generation half, Context Precision and Context Recall on the retrieval half. Classic n-gram scores such as BLEU and ROUGE measure overlap with a reference string and ignore retrieved context, which is why RAG-native meters exist. The wider family list, including those lexical scores, sits in every metric in common use. Synthetic question generation is a Ragas feature named on GitHub and walked in Vectara’s evaluating-RAG post; the bias and review workflow for that silver data live on Generating Synthetic Evaluation Data for RAG.

What do the four core Ragas metrics measure?

Ragas measures four core dimensions that Atlan, GeeksforGeeks, Medium (DhanushKumar, 24 September 2024), and Vectara all list in the same family: faithfulness, answer relevancy, context precision, and context recall. Higher is better on the usual 0 to 1 reporting scale. Answer relevancy is computed from embeddings and cosine similarity, so the numeric bound is a convention of the walkthroughs rather than a mathematical guarantee. The four scores separate retriever failures from generator failures instead of collapsing the pipeline into one vibe check.

Faithfulness measures whether claims in the generated answer can be inferred from the retrieved context. DhanushKumar’s Medium walkthrough (24 September 2024) uses the published Einstein case: question “Where and when was Einstein born?”, context stating birth on 14 March 1879 in Germany, and a low answer that keeps Germany but moves the date to 20 March. Statement one (“born in Germany”) is supported. Statement two (“born on 20 March 1879”) is not. Faithfulness is 1 supported claim out of 2, which is 0.5. Vectara’s evaluating-RAG post states the same rule in one sentence: if every claim in the answer can be inferred from the given context, the response is faithful to that context.

Answer relevancy (also called response relevancy in later API names) measures whether the answer addresses the user question. The Medium walkthrough defines it as the mean cosine similarity between the original user_input and N artificial questions reverse-generated from the response, with default N=3. Incomplete or padded answers score lower because the reverse questions drift from the original ask. Context precision measures whether relevant chunks sit high in the retrieved ranking: the mean of precision@k across ranks, with a relevance indicator at each rank (Medium). Context recall measures whether the retrieved contexts cover what a labelled reference answer needs: each claim in the reference is checked for attribution to the retrieved window, and the metric always requires that reference (Medium). GeeksforGeeks’s implementation sketch stores the sample as user_input, retrieved_contexts, response, and reference before calling evaluate().

How faithfulness and relevancy are computed across tools, and why two libraries can disagree on the same answer, is owned with the generation-metric definitions. Worked information-retrieval formulas for precision and recall sit with the retrieval-metric definitions. The matrix that follows is the Ragas-specific read: each metric beside the assumption that makes the number usable. Cross-tool generation depth is what each generation metric actually computes. Ranked-list arithmetic is information-retrieval metrics.

What does each Ragas metric assume about judges and labels?

Each Ragas metric returns a number only under the assumption that rides with it. A high faithfulness score proves consistency with the retrieved window, not that the window was worth retrieving. Atlan states the failure in one line: a RAG system can score 0.95 faithfulness and still produce wrong business answers if the retrieved content is stale or incorrect. The table pairs what you get with that ceiling so the dashboard number cannot be misread as world truth.

The two rows that surprise teams first are faithfulness without truth and recall without labels. Faithfulness can look excellent while the index is wrong. Recall cannot run until someone wrote the reference the metric treats as a proxy for needed contexts. Building and keeping that labelled file honest is golden test set. Synthetic candidates stay silver until a human verifies them; that review workflow is the synthetic-data sibling already linked above.

What does Ragas assume about your judge model and your labels?

Most Ragas metrics are LLM-assisted. Atlan groups Ragas with TruLens and DeepEval as frameworks that use LLM-as-a-judge for reference-free scoring at the inference layer. Vectara’s evaluating-RAG post is explicit that Ragas faithfulness is computed with an LLM-as-a-judge approach. The score therefore inherits that judge’s biases, temperature, and run-to-run variance. Atlan’s cost line is equally plain: the library is free, and evaluation cost comes from the model API calls used as judges, especially GPT-4-class judges on high-volume runs.

Field requirements split cleanly. Faithfulness and answer relevancy can run without a labelled reference because they compare the answer to retrieved context and to the question. Context recall always needs a reference, because recall is defined against what should have been retrieved, and the Medium walkthrough uses the reference answer as a proxy for those reference contexts. Reference-based precision variants and answer-correctness style scores also expect ground truth when you enable them. Atlan’s glance table still calls Ragas “reference-free by default” for the suite as a whole; that default does not erase the labelled requirement on recall.

Vectara contrasts the LLM-judge path with its Hughes Hallucination Evaluation Model (HHEM): a classification model for factual consistency that Vectara argues is more reliable than a general LLM for that single check. That contrast is a judge-design choice, not a claim that one class name ships inside every Ragas install. Measured failure modes of LLM judges, including position bias, verbosity bias, and self-preference, are catalogued at where it fails. Pin the judge model version, set temperature near zero for regression work, and re-baseline when either changes, or next week’s faithfulness average is not comparable to last week’s.

Where does Ragas disagree with human judgement?

A Ragas score is an LLM judge’s output on the inputs you passed. It can diverge from a human rater who knows the corpus, and from a human who knows the world, without any spreadsheet bug. The disagreement is the metric doing what it was designed to do: score consistency with retrieved context and with the question, not certify business truth.

Atlan documents the business case directly. A pipeline can post 0.95 faithfulness and still answer from stale or incorrect retrieved content. Independent benchmarking that Atlan cites (AIMultiple, same article) found a universal blind spot across tools: no framework correctly distinguished factually wrong contexts from factually correct ones under entity-swapped hard negatives. A passage with the right entities and the wrong answer outscored a passage with the right topic but no answer. That result is about context-relevance scoring under adversarial retrieval, not a Ragas-only faithfulness bake-off, and it still lands on the same lesson: inference-layer eval assumes the index was worth querying.

Two human-facing patterns follow from the faithfulness definition itself. High faithfulness on the wrong retrieved chunks looks good to the metric and bad to a reviewer who knows which document should have been fetched: the answer is faithful to garbage. A true fact that never appeared in the retrieved context fails faithfulness even when a world-knowledge human would accept it, because Ragas is scoring support from what was retrieved, not encyclopedic correctness (Medium faithfulness definition; Atlan grounding language; Vectara factual-consistency framing). Cross-tool predicate disagreement, where one library treats support as entailment and another treats it as non-contradiction, belongs with the generation-metric sibling already linked above. The scored pick among Ragas, DeepEval, TruLens, Phoenix, and Braintrust is scored on the same pipeline.

Is Ragas open source, and how does it compare to DeepEval?

Yes. Ragas is open source and free to install. Atlan’s April 2026 table lists the license as open source, setup as low and quick, and cost as free library plus judge tokens. The GitHub README and ragas.io both lead with pip install ragas. GitHub showed about 15.8k stars on the 18 September 2026 capture; Atlan’s earlier table listed 13.3k. Star counts move; treat them as popularity snapshots, not quality scores.

DeepEval is the tool Ragas is most often weighed against in the same Atlan comparison. Atlan places Ragas as RAG-pipeline focused with the four core metrics, limited custom-metric surface, and manual CI setup. It places DeepEval as a broader metric library (50+ metrics across RAG, agents, multi-turn, safety, and image), with native Pytest integration and Confident AI as the commercial cloud. Both are open source. Both bill the judge model separately. Atlan’s workflow advice is practical rather than crowning: choose Ragas when the primary question is RAG quality with a lightweight harness; choose DeepEval when the stack has matured into multi-component systems and CI ownership is shared with QA.

Which of the two wins for a given pipeline is not a profile verdict. Crowning either would falsify the neutrality the tools survey is built for. Weaviate publishes the metric definitions and the Ragas conversation that informed its evaluation posts; it does not replace Ragas or DeepEval as the thing that scores a run. The five-tool survey that keeps Ragas, DeepEval, TruLens, Phoenix, and Braintrust on one grid is the tools comparison already linked above.

Ragas is how many teams first put numbers on retrieval and generation without waiting for a full labelled suite. The four core metrics separate stages. Each metric carries an assumption about judges and labels. Faithfulness can look excellent on stale retrieval, and context recall will not run without a reference. Read every score as judge-mediated consistency with the inputs you passed, calibrate against humans on your own domain, and pick the install shape on the tools survey rather than by star count.