Public RAG Benchmarks and What They Actually Test

Public RAG benchmarks are fixed question, corpus, and label suites that score a retrieval-plus-generation pipeline under a published protocol, a named capability on someone else’s data, not a ship gate on yours. MIRAGE, RAGBench and CRAG freeze different stages. IBM Research’s MTRAG freezes multi-turn chat. Vectara’s Open RAG Benchmark freezes PDFs that mix text, tables and images. Evidently AI’s survey of seven RAG benchmarks (Dasha Maliugina, updated 19 May 2026) closes on the same caveat: benchmarks help compare models, and your system still needs custom evaluations on your own data. How those scores become a launch decision is evaluate retrieval and generation separately. What each of those meters computes is listed under every metric in common use.

Why does a RAG leaderboard rank rarely transfer to your corpus?

A public-benchmark rank transfers only when your corpus domain, query distribution, chunking and index setup, and scoring protocol match the bench, and they almost never do. Domain mismatch is the first failure. Wikipedia, web pages and product manuals are not internal PDFs, tickets or contracts. Onyx’s EnterpriseRAG-Bench page (captured 18 September 2026) is explicit that it is built on company-internal data, not Wikipedia: 511,962 documents across nine sources and 500 questions. A system that ranks well on open-domain factoids can still miss the vocabulary and clutter of that pile.

Query-type mismatch is the second. Single-hop fact lookup is a different job from multi-hop reasoning or a global question that needs themes across a whole collection. Microsoft Research’s BenchmarkQED post (Darren Edge, Ha Trinh, Andres Morales Esquivel and Jonathan Larson) separates local queries, answers found in a few text regions, from global queries that require reasoning over large portions of the dataset. AutoQ synthesises four query classes along that spectrum so a vector retriever is not scored only on the neighbour-lookup job it already wins. Pipeline confound is the third: a published score freezes someone else’s chunker, retriever, top-k and generator. Change any of those on your side and the leaderboard number no longer isolates what you changed.

Judge mismatch is the fourth. Exact match and F1, LLM-as-a-judge prompts, and TRACe-style span labels are not interchangeable. Two benches can both say faithfulness and still disagree on the same answer. NVIDIA’s RAG-blueprint accuracy doc (captured 18 September 2026) scores end-to-end answer accuracy with the NVIDIA Answer Accuracy metric from Ragas, Mixtral-8x22B-Instruct-v0.1 as judge, on a 0 to 4 scale later normalised. That is a different instrument from CRAG’s perfect / acceptable / missing / incorrect labels with auto-eval at +1, 0, and -1. The family of those meters is not a benchmark catalogue. Bias in the judge itself is where it fails. What each named suite actually freezes is the next question, because a transfer failure is only diagnosable once you know the stage.

What does each public RAG benchmark actually test?

Each public RAG benchmark tests one frozen capability on one frozen corpus under one frozen protocol. Evidently AI’s May 2026 survey names seven: Needle-in-a-Haystack, BEIR, FRAMES, RAGTruth, RULER, MMNeedle and FEVER. RAGBench, CRAG, IBM MTRAG, Vectara Open RAG Benchmark, Onyx EnterpriseRAG-Bench, Microsoft BenchmarkQED, and NVIDIA’s blueprint runs on named public sets sit on the same captured set. MIRAGE is the compact QA pool Park et al. published for retriever-generator pairing. Sizes and metric names below come from those sources. No leaderboard score is invented.

Vendor platform boards sit beside those suites and are not substitutes for them. NVIDIA’s blueprint doc reports its own stack, nvidia/llama-3.3-nemotron-super-49b-v1.5 as generator, on FinanceBench, HotpotQA, Google Frames and ViDoRe-v3 subsets; Google Frames moved from 0.486 with reasoning off to 0.597 with reasoning on under that judge. Contextual AI’s 15 January 2025 platform post reports 71.2 percent on RAG-QA Arena against a Cohere-plus-Claude-3.5-Sonnet baseline at 66.8 percent, and a BEIR reranker score of 61.2 against Voyage-v2 at 58.3 across 14 datasets. Those are named vendor runs. They are not a protocol you can rerun on a different stack without their pipeline. Dedicated depth on RAGBench, CRAG and MIRAGE follows because the title promises those three by name, and because the papers spell the stage each one actually scores.

What does RAGBench measure?

RAGBench (Robert Friel, Masha Belyi and Atindriyo Sanyal, Galileo Technologies, arXiv 2407.11005v2, 16 January 2025) measures explainable RAG evaluation, not only whether the final string matches a reference. The dataset holds 100k examples across five domains, bio-medical research, general knowledge, legal contracts, customer support and finance, assembled from 12 component datasets: PubMedQA, CovidQA-RAG, HotpotQA, MS Marco, HAGRID, ExpertQA, CUAD, DelucionQA, EManual, TechQA, FinQA and TAT-QA. Context token length ranges from about 100 to 11k, with CUAD at the long end. Labels are released on Hugging Face as rungalileo/ragbench. The paper is one of the papers that matter for evaluation, not a retrieval-only leaderboard.

TRACe is the metric suite: utilization, relevance, adherence and completeness. Relevance is the fraction of retrieved context that is useful for the query. Utilization is the fraction the generator actually used. Completeness is how much of the relevant span made it into the answer. Adherence is the hallucination flag, synonymous in the paper with faithfulness, groundedness and attribution. Component sets contain between 1 percent and 20 percent hallucinations, with ExpertQA, CovidQA and MS Marco at 12, 16 and 13 percent and CUAD, FinQA and TAT-QA at about 1 percent each. GPT-4 as annotator reached 93 percent example-level and 95 percent span-level agreement with human judgements on DelucionQA adherence.

The headline experimental claim is that a fine-tuned DeBERTa-v3-Large NLI model, about 400 million parameters with a head per TRACe metric, outperforms few-shot LLM judges on hallucination AUROC across most domain slices, with DeBERTa AUROC from 0.64 to 0.86 depending on the subset. RAGAS and TruLens are the zero-shot baselines in that table; they are harnesses, not RAGBench. Named evaluation stacks are scored on the same pipeline. CRAG is the complementary end-to-end factual suite on the same ranking set, and it scores answers rather than training an evaluation model.

What does CRAG measure?

CRAG, Comprehensive RAG Benchmark (Xiao Yang et al., Meta / KDD Cup 2024, arXiv 2406.04744), measures open-domain factual question answering when the system may call mock web and knowledge-graph search APIs, not only an index you already built. The GitHub README (facebookresearch/CRAG, captured 18 September 2026) states five domains and eight question categories, with entity popularity from popular to long-tail and temporal dynamism from years to seconds. The dataset schema and mock APIs live in that repository. Scoring rates each response as perfect, acceptable, missing or incorrect. Auto-evaluation assigns correct at 1 point, missing at 0, and incorrect at -1, mixing rule-based matching and an LLM check. That -1 is the production tell: a fluent wrong answer is worse than a refusal.

Use CRAG when the claim under test is open-domain factual RAG with tool-like retrieval. Do not use it as a ship gate for a private PDF corpus with no web API. RAGBench’s related-work section is explicit that CRAG, ChatRAGBench, DomainRAG and ALCE are ground-truth response suites scored with exact match or ROUGE, which is a different job from TRACe labels. The two papers are not substitutes. One trains an evaluation model. One scores an end-to-end factual agent against mock search.

Three baselines ship in the repo for demonstration. Do not treat a demo baseline as a published SOTA for your stack. License on the migrated repository is CC BY-NC 4.0, so a commercial rerun needs a separate reading of that file. MIRAGE is the compact retriever-plus-generator pool the title still owes, because a 37,800-chunk index is the opposite of CRAG’s mock web.

What does MIRAGE measure?

MIRAGE (Chanhee Park, Hyeonseok Moon, Chanjun Park and Heuiseok Lim, Korea University, arXiv 2504.17137) measures retriever and generator together on a compact pool so you are not indexing all of Wikipedia to score a pairing. The abstract states 7,560 curated instances mapped to a retrieval pool of 37,800 entries. Each query has at least one positive chunk with the critical information and several negatives that are similar but lack the key span. Construction reorganised PopQA, Natural Questions, TriviaQA, IfQA and DROP; HotpotQA-style multi-hop sets were excluded to keep the job single-hop. Chunks are 330 tokens via BERT-base-uncased. The enwiki dump used for mapping is dated 1 September 2024.

The four RAG adaptability metrics are noise vulnerability, context acceptability, context insensitivity and context misinterpretation. They score whether the generator uses relevant context, ignores noise, and does not invent when the window is empty or wrong. That is a pairing test. A high generator score with a weak retriever, or the reverse, shows up as a metric rather than as a single exact-match. Code and data are pointed at github.com/nlpai-lab/MIRAGE. Name collision: MIRAGE-Bench (Thakur et al., arXiv 2410.13716) is a different multilingual arena; Park et al. 2504.17137 is the compact English pool. Multilingual retrieval depth is multilingual RAG.

Do not read a MIRAGE pairing as a prediction for your tenant. The pool is Wikipedia chunks with planted hard negatives, not Confluence plus Slack. Embedding-model selection on public retrieval boards is a different decision, owned by how to choose an embedding model for RAG. Choosing among these suites, including the ones that are not MIRAGE, is the practical next question, because running all of them is not a method.

How do you choose which RAG benchmark to run?

You choose the benchmark that answers the question you are actually asking, not the one with the loudest leaderboard. Retrieval rank quality alone is BEIR. Long-context needle finding is NIAH or RULER. Hallucination types on RAG responses are RAGTruth. Multi-hop Wikipedia reasoning is FRAMES. Claim verification is FEVER. Image-window needles are MMNeedle. Explainable industry-domain labels are RAGBench and TRACe. Compact retriever-plus-LLM adaptability is MIRAGE. Open-domain factual RAG with mock search is CRAG. Multi-turn enterprise chat, including “I don’t know,” is IBM MTRAG: 110 conversations across finance, general knowledge, IT documentation and government pages, with many turns written to be unanswerable. Multimodal PDFs are Vectara’s Open RAG Benchmark: 1,000 arXiv papers, 3,000 or more question-answer pairs, extractive and abstractive, text plus tables plus images, in BEIR format.

Company-internal clutter is the job Onyx named. EnterpriseRAG-Bench publishes 511,962 documents, 500 questions and a 12-system board. Onyx’s overall bars, captured 18 September 2026, list Weaviate Verba at 34.5, LlamaIndex at 27.2, LangChain at 25.0 and Open WebUI at 24.9, with OpenClaw at 68.2 at the top of a list they sorted by score. Those figures are Onyx’s published overall percentages on their synthetic-company corpus, not an independent bake-off and not a Weaviate product score on your data. NVIDIA’s blueprint numbers are the same class of evidence: a named stack on named public sets, Mixtral-8x22B as judge, FinanceBench 0.612 with reasoning off and 0.668 with reasoning on, RAGBattle 0.867 with VLM on and reasoning off. Attribute them to that doc. Do not carry them into a launch email as if they were your corpus.

Microsoft BenchmarkQED is the odd one out: it is a harness for your documents, AutoQ for query classes, AutoE for pairwise LLM judging on comprehensiveness, diversity, empowerment and relevance, AutoD for sampling to a target breadth and depth. On 1,397 AP News health articles they generated 50 queries per class, 200 total, and compared LazyGraphRAG to GraphRAG, vector RAG at 8k and 120k, LightRAG, RAPTOR and TREX, then to a 1-million-token vector window. Win rates are theirs. The transferable fact is the local-versus-global split, not a number to copy. If none of these suites match your documents and query mix, a mismatched public bench is worse than no bench, and the ship gate moves to a labelled file you own.

When should you ignore public benchmarks and build your own test set?

Ignore public leaderboards as a ship gate whenever your corpus domain or query mix is not represented, which is the default for enterprise internal knowledge. Public benches remain useful as component smoke tests, a retriever on BEIR, a long-context model on NIAH or RULER, a hallucination detector on RAGTruth, a PDF parser on Vectara’s 1,000 papers, and as regression oracles for published methods. Evidently AI states that close in one sentence: while benchmarks help compare models, your RAG system needs custom evaluations on your own data. Microsoft built BenchmarkQED because GraphRAG’s global queries are not what a nearest-neighbour freeze measures. IBM built MTRAG because one-off QA is not a live chat.

The production gate is a labelled file on your documents: questions you care about paired with spans you trust. Build that file with the method at a golden test set. Expand coverage carefully with synthetic evaluation data. Once traffic exists, fold in Evaluating RAG in Production. Onyx’s 500-question internal set is an existence proof that Wikipedia boards miss Slack-and-Drive clutter; it is still their synthetic company, not yours. Vectara’s hard-negative PDFs test whether retrieval ignores irrelevant papers; they are still arXiv. A frozen silver file from last quarter is last quarter’s product.

Run a public suite when you are choosing a component the suite actually isolates. Run your own file when you are choosing to ship. Do not average a BEIR NDCG, a CRAG auto-eval and an Onyx overall bar into one health score. They do not share a judge, a corpus or a question mix. Split retrieval from generation on your labels the same way the evaluation hub splits them, or a high MIRAGE pairing will hide a CRAG-style -1 on the questions your users actually ask.

Public RAG benchmarks are frozen protocols on someone else’s data. Evidently’s seven, RAGBench’s 100k TRACe labels, CRAG’s +1/0/-1 factual API, MIRAGE’s 7,560 by 37,800 adaptability pool, IBM’s 110 conversations, Vectara’s 1,000 PDFs and Onyx’s 511,962-document internal set each answer a different question. A rank transfers only when domain, query class, pipeline freeze and judge match. Use the suite that isolates the component you changed. Ship on a labelled file of your own.