Public RAG Benchmarks and What They Actually Test

Public RAG benchmarks are fixed suites scoring retrieval-plus-generation under a published protocol. They measure a named capability on someone else’s data — not readiness on yours. MIRAGE, RAGBench and CRAG test different stages; leaderboard ranks rarely transfer. Evaluation · metrics.

Why does a RAG leaderboard rank rarely transfer to your corpus?

A public-benchmark rank transfers only when your corpus domain, query distribution, chunking and index setup, and scoring protocol match the bench — and they almost never do. Four failures explain most surprises:

• Domain mismatch. Wikipedia, web pages and product manuals are not your internal PDFs, tickets or contracts. A system that ranks well on open-domain factoids can still miss the vocabulary and structure of your corpus.
• Query-type mismatch. Single-hop fact lookup is a different job from multi-hop reasoning or “global” questions that need themes across a whole collection. Microsoft Research’s BenchmarkQED work (2025) separates local queries (answer in a few regions) from global ones (reason over large portions of the dataset) for exactly this reason.
• Pipeline confound. A published score freezes someone else’s chunker, retriever, top-k and generator. Change any of those on your side and the leaderboard number no longer isolates what you changed.
• Judge mismatch. Exact match and F1, LLM-as-a-judge prompts, and TRACe-style span labels are not interchangeable. Two benches can both say “faithfulness” and still disagree on the same answer.

Evidently AI’s own RAG-benchmarks survey (updated May 2026) closes on the same caveat: benchmarks help compare models, but your RAG system needs custom evaluations on your own data. The honest ship gate is a golden set on your documents — see building a golden test set — not a screenshot of someone else’s leaderboard.

What does each public RAG benchmark actually test?

The table below is the transfer-risk map: what stage each named public suite actually stresses, what corpus it was built on, which metric family it reports, and why a high rank still fails to predict your production quality. Sizes and metric names come from the cited papers or project READMEs; no leaderboard scores are invented here.

Quote two or three benches that cover different stages — retrieval, generation faithfulness, and your actual query mix — rather than one headline number. Embedding-model selection via MTEB is a related but separate decision at how to choose an embedding model.

What does MIRAGE measure?

MIRAGE (Park, Moon, Park and Lim, Korea University, arXiv:2504.17137, April 2025) is a compact RAG question-answering benchmark built so you can score both the retriever and the generator without indexing a full Wikipedia dump. The release pairs 7,560 curated instances with a retrieval pool of 37,800 document chunks: each query has at least one positive chunk that contains the answer and several hard negatives that look similar but lack the key fact.

The paper’s distinctive contribution is four RAG adaptability metrics — noise vulnerability, context acceptability, context insensitivity and context misinterpretation — that probe whether the generator uses relevant context and ignores distractors, not only whether the final string matches a gold answer. The dataset is reorganised from PopQA, Natural Questions, TriviaQA, IfQA and DROP; articles were chunked at 330 tokens; multi-hop sets such as HotpotQA were deliberately excluded so the bench stays single-hop.

What does RAGBench measure?

RAGBench (Friel, Belyi and Sanyal, Galileo Technologies, arXiv:2407.11005) is a roughly 100,000-example RAG dataset assembled to train and benchmark RAG evaluation models — systems that score other RAG pipelines — not only to report end-to-end exact match. It spans five industry-oriented domains (bio-medical research, general knowledge, legal contracts, customer support, finance) drawn from twelve component sources including PubMedQA, CovidQA, HotpotQA, MS Marco, CUAD, DelucionQA, EManual, TechQA, FinQA, TAT-QA, ExpertQA and HAGRID. Context lengths in the suite range from about 100 tokens to about 11,000 tokens on long legal contracts. The labelled set is released on Hugging Face as rungalileo/ragbench.

Alongside the data, the authors formalise TRACe: Utilization (how much retrieved context the generator actually used), Relevance (retriever quality vs the query), Adherence (faithfulness / groundedness of the answer) and Completeness. Those labels are meant to be actionable — they point at which stage failed — in a way a single exact-match score cannot. Treat RAGBench as a shared yardstick for explainable RAG judges and cross-domain comparison, not as proof that your private corpus pipeline is ready. Where LLM judges themselves fail is covered at LLM-as-a-judge.

What does CRAG measure?

CRAG — the Comprehensive RAG Benchmark (Yang et al., Meta, arXiv:2406.04744; KDD Cup 2024 shared task) — is a factual question-answering suite that ships question-answer pairs and mock APIs simulating web and knowledge-graph search. The design covers five domains and eight question categories, with entity popularity from head to long-tail and temporal dynamism from years down to seconds (project README, facebookresearch/CRAG).

Responses are graded perfect, acceptable, missing or incorrect. The published auto-evaluation assigns +1 for correct, 0 for missing and −1 for incorrect, combining rule-based matching with LLM assessment. Reach for CRAG when the claim under test is open-domain factual RAG with tool-like retrieval. Skip it as a ship gate when your system only searches a private document set with no web or KG API in the loop — that mismatch is exactly the transfer failure the opening section names.

How do you choose which RAG benchmark to run?

Pick the benchmark that answers the question you are actually asking — not the one with the loudest leaderboard. A practical map:

• Choosing an embedding model → MTEB (with BEIR as the retrieval-only check) — embedding model selection.
• Measuring retrieval rank quality alone → BEIR (NDCG@10 across diverse IR tasks).
• Stressing long-context needle finding → Needle-in-a-Haystack or RULER.
• Measuring hallucination on RAG responses → RAGTruth (and faithfulness metrics on your own set).
• Explainable industry-domain RAG labels → RAGBench / TRACe.
• Compact retriever + LLM adaptability with hard negatives → MIRAGE (arXiv:2504.17137).
• Open-domain factual RAG with mock search APIs → CRAG.
• Multilingual RAG generation → MIRAGE-Bench (Thakur et al.) — then the mechanism page at multilingual RAG.
• Comparing methods on your own documents → BenchmarkQED (AutoQ / AutoE / AutoD) or a hand-built golden set.

If none of those match your documents and query mix, the next section is the decision — not another public leaderboard. Frameworks that score your pipeline (Ragas, DeepEval, TruLens and the rest) are compared at evaluation tools; they are harnesses, not public benchmarks.

When should you ignore public benchmarks and build your own test set?

Ignore public leaderboards as a ship gate whenever your corpus domain or query mix is not represented on the bench — which is the default for enterprise internal knowledge. Public suites remain useful as component smoke tests (a retriever on BEIR; long-context behaviour on NIAH) and as regression oracles when you reproduce a published method. They are not a substitute for questions you care about paired with spans you trust on your documents.

Build that golden set with the method at building a golden test set; expand coverage carefully with synthetic evaluation data; and once traffic exists, fold in online evaluation. That is the path from a published rank to a number that can actually block a bad deploy.