Generating Synthetic Evaluation Data for RAG

Synthetic evaluation data for RAG is a machine-generated set of question, context, and answer triples built from your corpus so you can score retrieval and generation without waiting on a fully human-labelled set. It is silver until a human verifies it, not a replacement for a golden test set. Weaviate’s overview of RAG evaluation (Erika Shorten and Connor Shorten, 21 November 2023) records that ARES, a framework that trains lightweight judges on synthetic queries, still requires a human preference validation set of 150 annotated datapoints or more plus five few-shot in-domain queries. Ragas’s v0.1.21 testset-generation docs (re-fetched 18 September 2026) claim synthetic generation can reduce developer time in data aggregation by 90 percent; that 90 percent is a Ragas vendor figure, not an independent benchmark. How those scores become a launch decision is evaluate a RAG system.

Why generate synthetic RAG test questions?

Synthetic RAG test questions exist because labelled domain triples are slow and expensive, especially on private or regulated corpora, while knowledge bases drift faster than humans can relabel. Red Hat Developer (William Caban Babilonia, 23 February 2026) puts the production failure in one sentence: most RAG systems reach production with weak evaluation, tuning embeddings and prompts while still relying on manual spot checks, small hand-labelled sets, or generic LLM-as-a-judge metrics. Without known ground-truth context, a failure is hard to assign to retrieval versus generation. Synthetic triplets give you that context on purpose.

Three jobs dominate what teams use the data for. Repeatable benchmarks compare embedding models, chunking strategies, and LLM configurations over time, which is Red Hat’s listed reason to generate the set at all and the job that belongs in regression testing. Offline experimentation uses the same file to A/B retrievers and prompts before traffic sees them. Drift priming pairs synthetic breadth with sampled production queries so the eval set tracks real language, not only generator style; mining that traffic is Evaluating RAG in Production. NVIDIA’s synthetic-data pipeline post (captured 18 September 2026) adds a fourth, narrower job: generating queries whose difficulty you can filter, so a FiQA-style retrieval comparison is not run only on easy extractive questions. Even after that filter, MiniLM recall@5 sits at 0.4958 on synthetic FiQA versus 0.4636 on the human benchmark, which is why a synthetic-only average still needs a human calibration slice.

Ragas’s own documentation is explicit that human-generated questions may struggle to reach the complexity a thorough evaluation needs, which is why it evolves simple questions into reasoning, conditioning, and multi-context forms. That is a coverage argument, not a licence to skip review. Microsoft’s golden-dataset post still needed a finance expert to fail 27 of 100 silver samples. Synthetic breadth without that pass is still silver. Red Hat’s “automatic ground truth” claim is true only of the cited span the pipeline extracted, not of world knowledge. If the source document is wrong, the synthetic answer is faithfully wrong, and every downstream faithfulness score will call that a pass.

How is synthetic RAG test data generated?

Synthetic RAG test data is generated by grounding a model in your documents, drafting questions from those documents, then filtering pairs that are not answerable from the cited span. Red Hat’s SDG Hub flow is six named stages: topic extraction, question generation, question evolution into user-style queries, answer generation grounded in source content, groundedness filtering that drops low-quality pairs, and context extraction of the minimal gold span. The output contract is a question, an answer, and gold context designed so retrieval and generation can be scored separately.

Ragas’s current getstarted path (docs re-fetched 18 September 2026) does the same job with a knowledge graph. Documents become nodes, default_transforms enrich the graph, and TestsetGenerator samples scenarios from that graph. On Ragas’s published sample, the graph grows from 10 document nodes to 48 nodes and 605 relationships after transforms. The default query distribution, printed by default_query_distribution, is SingleHopSpecificQuerySynthesizer at 0.5, MultiHopAbstractQuerySynthesizer at 0.25, and MultiHopSpecificQuerySynthesizer at 0.25. The tutorial call is generate_with_langchain_docs with testset_size=10, a demo size, not a published production budget. Weaviate leads the named-library list here because its RAG-evaluation post is the ARES pairing; Ragas, Red Hat SDG Hub, Amazon Bedrock, and NVIDIA all ship generators that start from documents you provide.

Amazon’s Bedrock post (captured 18 September 2026) names critique agents that score generated questions for relevance and generated answers for groundedness before the pair is kept. NVIDIA inserts an embedding-model-as-judge after generation: cosine similarity between question and context is thresholded so easy paraphrase questions can be dropped. Those filters are the mechanism. AWS also walks a notebook that loads a corpus, generates pairs, and keeps only pairs the critique agents accept; the GitHub sample that ranks beside that post is the same pipeline in a notebook, not a second method. Full install walkthroughs stay with the vendors. The library that owns Ragas’s assumptions is Ragas explained. Tool choice among named stacks sits with evaluation tools compared. Which question types those generators actually emit is the next constraint, because a simple-only set inflates every score that follows.

What question types do synthetic generators produce?

Useful synthetic sets mix easy extractive questions with harder multi-hop, reasoning, conditional, and conversational forms. Uniform simple-only sets inflate scores because every retriever looks good when the question is a near-copy of one chunk. Ragas’s v0.1.21 evolution list names reasoning (rewrite to need inference), conditioning (add a conditional element), multi-context (need more than one section), and conversational follow-ups that mimic chat. The newer synthesizer names, SingleHopSpecific, MultiHopAbstract, and MultiHopSpecific, are the same coverage idea with different labels.

NVIDIA’s pipeline treats difficulty as a filter you set, not a side effect. An embedding judge ranks question-context cosine similarity; lower thresholds keep a higher share of hard questions. On FiQA, they compared generated queries against the public FiQA benchmark for five to six embedding models and chose the 70th-percentile threshold because it yielded the lowest average deviation in retrieval metrics, 4.57 percent on recall@5, versus 5.65 percent at the 75th percentile. Correlation of model ranking between synthetic and human FiQA queries was 0.91 at both thresholds. The published recall@5 rows include all-MiniLM-L6-v2 at 0.4636 on human FiQA versus 0.4958 synthetic at the 70th percentile, e5-large-v2 at 0.52727 versus 0.54622, and sentence-t5-large at 0.5091 versus 0.51261. Those figures are NVIDIA’s published FiQA analysis, with nv-embedqa-e5-v5 as the embedding judge, not a site-wide difficulty law. They repeated the deviation check on Natural Questions and internal IT datasets and kept the filter only where average deviation of retrieval metrics stayed under 10 percent.

Prompt Engineering Guide’s synthetic RAG page and Galileo’s generation post (both captured 18 September 2026) add persona and adversarial phrasings as coverage the production distribution will eventually hit: negation, out-of-corpus refusals, and questions that look in-domain but are unanswerable. Galileo treats generation from your own documents as the way to avoid leaking public-benchmark items into a private eval. Unanswerable probes belong in the golden file this silver feeds. A set that never asks a question the corpus cannot answer will score a system that always answers as if it were honest. Bias in how those questions are written is a separate failure from missing a type.

What bias does synthetic RAG evaluation data carry?

Synthetic RAG evaluation data often underestimates difficulty and can favour the generator model’s style, so a set that ranks retrievers usefully can still mislead generator comparisons. NVIDIA’s FiQA table is the honest version of that split: recall@5 on synthetic queries tracked human FiQA ranking (correlation 0.91) once the easiness filter was calibrated, which is a retrieval result. It does not say the same calibration would hold if you used the same file to crown a generator architecture. Ragas’s own docs warn that LLMs by default follow common paths and are not good at creating diverse samples unless you force evolution. That is stylistic collapse: the generator writes questions the way it writes answers.

Wording leakage is the mechanical version. If the question copies a unique phrase from the chunk, lexical retrievers win by string match and you never tested whether a user paraphrase would retrieve. Amazon’s critique agents exist to catch question-irrelevance and ungrounded answers; they do not catch a question that is a near-clone of the context. Dev.to’s 2025 curation post (Kuldeep Paul, captured 18 September 2026) treats near-duplicate pruning and drift-aware splits as first-class, because an un-deduplicated silver set double-counts one intent and under-counts another. The privacy paper “Diverse And Private Synthetic Datasets Generation for RAG” (arXiv 2508.18929v1) is the enterprise limit: corpus text can leak through generated questions unless generation is treated as a privacy problem, not only a coverage problem. Full differential-privacy pipelines stay with that paper.

Weaviate’s ARES write-up is the reason a human seed still sits under the synthetic pile: ARES generates a large synthetic query set from few-shot examples, then filters with round-trip consistency (can we retrieve the document that produced the query?), and still needs 150 or more human-annotated datapoints to train the judges. Weak negatives are random chunks from other documents; strong negatives are a chunk from the same document or a BM25 top-10 neighbour. Synthetic volume does not remove the labelled core. Judge bias on those generated labels is owned by LLM-as-a-Judge for RAG. Public leaderboards that never saw your corpus are RAG benchmarks. Validation gates are how you keep the remaining bias from becoming a ship number.

How do you validate synthetic RAG test questions?

Ungated synthetic triples miscalibrate scores. Every production-bound set needs a groundedness or answerability filter, a difficulty or diversity check, and a human spot-check before it is treated as gold. Red Hat’s groundedness filtering stage is the drop-low-quality step. Amazon’s critique agents score question relevance and answer groundedness. NVIDIA publishes two measured gates in that pipeline post. For answerability, human annotators scored generated queries A through D on whether the chunk answered the question: A if the chunk completely answered it, B if key information was missing, C if the query was relevant but the context did not answer it, and D if the query was not relevant. An LLM-as-judge prompt developed with those annotators, using Llama-3.1-70b-Instruct, reached 94 percent precision and 90 percent recall against that human protocol. For easiness, the embedding-judge threshold was chosen so average deviation of recall@5 versus human FiQA stayed under 10 percent; 4.57 percent was the published 70th-percentile result.

Those percentages are NVIDIA’s published filter quality on their protocol, not a universal accept rate. ARES’s round-trip check, as Weaviate summarises it, is a retrieval-shaped cousin: if you cannot retrieve the source document from the synthetic query, the pair is a weak evaluation item. Diversity still has to be inspected by hand or by embedding clusters, because a filter that only checks groundedness will keep 500 paraphrases of one FAQ. Human spot-check is the last gate. Microsoft’s 66 of 100 greenlight on a silver earnings-call set is the measured cost of skipping it; that figure lives on the golden-set node because that is where silver becomes gold.

Red Hat’s output is a clean triplet file: question, answer, gold context. Context extraction is the last stage because a long source document is not a retrieval label; the minimal span is. If you store only the parent file ID, you cannot tell whether the retriever returned the right section. That is the same stable-ID rule the golden-set node uses, applied one stage earlier. Ragas’s pandas export is the inspection step: generate more than you will keep, then drop rows that are duplicates, too easy, or not standalone. NVIDIA’s annotators gave no score at all when a query was not standalone, which is why independence sits beside groundedness in their protocol. Scoring the live pipeline against the surviving file is a different job. Faithfulness of generated answers is measuring generation. The labelled file you promote after these gates is where they come from. When the synthetic file is enough by itself, and when it is not, is the last decision, because a gated retrieval regression and a generator bake-off are not the same use.

When is synthetic data enough for RAG evaluation?

Synthetic data is enough for broad CI retrieval regressions when it is gated, task-aligned, and paired with a small human-reviewed core. It is not enough alone to crown a generator model or to replace production sampling. NVIDIA’s FiQA result is the retrieval case: after the easiness filter, synthetic ranking of embedding models correlated 0.91 with the human benchmark and deviated 4.57 percent on recall@5. Red Hat’s closed-loop workflow is the operational case: generate from the current corpus, score retrieval and generation separately, then feed failures back. Weaviate’s ARES requirement of 150 or more human labels is the calibration case: synthetic volume sits on top of a labelled seed, not instead of it.

Ragas’s 90 percent time-saved claim is about aggregation effort, not about replacing human judgement on the items you actually ship against. Prompt Engineering Guide and Galileo both treat synthetic generation as a bootstrap that still needs review. Kuldeep Paul’s curation post treats the dataset as living: production imports, synthetic variations, deduplication, and splits that track drift. A frozen silver file from last quarter is last quarter’s product. Deduplication across synthetic and production sources is the step most teams skip: if a support ticket and a generated question ask the same thing, keeping both inflates coverage counts and double-weights that intent in every average.

Use synthetic for breadth and for CI. Keep human gold for calibration anchors. Sample production traces for drift. The versioned gold file is what the build asserts. The generator that created the silver is not a source of truth about its own answers. If a chunking change drops recall@5 on the gated synthetic file, that is a retrieval regression worth blocking. If a new generator model scores higher on synthetic answers that it also wrote the questions for, that is self-preference in the data, not a quality win. Split the generator that builds the set from the generator you are scoring, the same way you split the judge family from the answer family.

Synthetic evaluation data is how you get question-context-answer triples at a scale humans will not label, and it stays silver until a groundedness filter and a human pass say otherwise. Ragas’s 90 percent is a vendor time claim. NVIDIA’s 0.91 FiQA correlation and 94 percent answerability precision are published filter results on named protocols. Weaviate’s ARES still wants 150 human labels under the synthetic pile. Generate for coverage, gate for honesty, and promote only what a reviewer will defend.