Building a Golden Test Set for RAG

A golden test set for RAG is a versioned collection of questions paired with reviewed answers and the sources those answers should come from, so retrieval and generation scores have labelled ground instead of vibes. Without it, Precision@k, Recall@k, faithfulness, and answer relevance have nothing trusted to compare against. Langfuse’s golden-dataset guide (captured 18 September 2026) calls that trust the entire value: when an experiment drops, you want the conclusion to be that the change regressed quality, not that item 23 has a wrong label. GetMaxim’s step-by-step (captured the same date) adds versioning, metadata, and expected outcomes so the file can serve pre-release evals and production observability. Statsig (31 October 2025) frames the same artifact as a shared reference that ends quality debates argued from feelings. How those scores become a launch decision is tool-neutral evaluation.

What is a golden test set for RAG?

A golden test set is a curated, trusted benchmark of cases your RAG pipeline must handle. Langfuse defines a golden dataset, also called a gold set or reference set, as a collection of test cases where each item records an input and, usually, a reviewed definition of what a correct response looks like. Golden means someone with domain knowledge confirmed the expected output, or deliberately decided the item does not need one because a reference-free check such as format or refusal is the pass criterion. An unreviewed pile of examples produces scores nobody acts on. A golden dataset produces scores that can block or promote a release.

Microsoft’s path to a golden dataset (Alexandra Savelieva and Sandip Kulkarni, Data Science + AI at Microsoft, 21 May 2024) starts from the same gap for RAG specifically. Unlike LLM evaluation, where public datasets exist, RAG projects are usually domain-specific and often sit on data that is not public. The prerequisite is a custom benchmark: a set of samples where each sample has a question, an answer, and optionally, but highly recommended, a reference to the source. Statsig’s 31 October 2025 perspective puts that custom file in organisational language: outputs get compared to known answers, gaps show up fast, and correctness, relevance, groundedness, and retrieval relevance stay the conversation instead of taste. GetMaxim’s definition is the operational envelope around the same core: a curated, versioned collection of prompts, inputs, contexts, and expected outcomes plus metadata, used across experimentation, simulation, evals, and in-production observability.

DataAspirant’s RAG evaluation harness (captured 18 September 2026) states the three-field minimum in code terms: the question is what runs through the pipeline, the reference is the ground truth judge-based metrics compare against, and the expected source powers retrieval metrics, with None marking unanswerable entries where the right behaviour is a refusal. A sloppy reference, in that same post, makes every downstream score noise. What each of those scores means sits with the complete catalogue. What actually goes in one record, including stable source IDs that survive re-chunking, is the next question, because a definition without a schema is still a slogan.

What goes in each RAG test case?

Each RAG test case needs a question, a reference answer you trust, and stable identifiers for the sources that should be retrieved, plus enough metadata to slice the set later. Microsoft’s benchmark structure is question, answer, and a source reference. DataAspirant stores those as question, reference, and expected_source, and uses None on expected_source for honesty probes the corpus cannot answer. Langfuse’s SDK item is input, expected_output, optional metadata, and optional source_trace_id so a reviewer can open the production trace that produced the case. GetMaxim’s metadata list is the slice layer: document IDs, timestamps, licenses, freshness windows, intent, persona, difficulty, language, safety category, reviewer, and risk tags.

Stable IDs have to name a document and section, not a transient chunk row. If you re-chunk the corpus, row 47 of last week’s index is a different span than row 47 of this week’s index, and every retrieval label that pointed at row 47 now lies. Langfuse’s JSON Schema on input and expected_output is the mechanical check for that contract: once set, every item is validated on creation, update, and CSV import, and invalid items are rejected with the validation error. Define the schema on day one. Retrofitting one onto a thousand inconsistent items is a cleanup project. The JSONL below is a schema illustration, not a published corpus. It keeps Microsoft’s three-part shape, DataAspirant’s unanswerable None, and GetMaxim-style source IDs that survive re-ingestion.

{"id":"q-014","question":"How long do I have to return a physical product?","reference_answer":"Physical items can be returned within 30 days of purchase.","relevant_source_ids":["refunds-policy#returns-window"],"answer_type":"factual","source":"real_support_ticket"}
{"id":"q-031","question":"Can I return a digital download after opening it?","reference_answer":"Digital downloads are not returnable once the download link has been used.","relevant_source_ids":["refunds-policy#digital"],"answer_type":"factual","source":"sme_authored"}
{"id":"q-088","question":"What is our store's policy on time travel refunds?","reference_answer":"I don't know. This is not covered in the corpus.","relevant_source_ids":[],"answer_type":"unanswerable","source":"honesty_probe"}

Retrieval metrics need those source labels the way Precision@k needs a relevant set; the formulas live with right chunk was ever fetched. Generation metrics still need the reference answer and the question, even when they do not look at source IDs. How many such rows you actually need is not one number, and published size guidance does not pretend it is.

How many questions do you need in a RAG test set?

There is no single published magic number for a RAG golden set. What published guides actually recommend, as of 18 September 2026, is a range matched to the job, then human review. Microsoft’s Savelieva and Kulkarni post treats about 100 QA samples as a reasonable number for diversity without overwhelming resources, and recommends generating about 100 questions, expanding to a few hundred with more resources. Those extra resources are needed twice: once to curate the benchmark, and again when the verified set is used to score end-to-end RAG. Their own case study generated 100 questions from Microsoft earnings-call transcripts, ten questions at each location per pair of randomly selected chunks.

Microsoft also publishes the cost of doing this by hand, which is why synthetic silver exists at all. In a recent three-week RAG prototype, subject-matter experts on the customer side took eight to ten days to come up with 10 question-answer samples in two cases, and about two weeks for approximately 90 QA pairs in a two-month engagement. After generation, one Microsoft finance expert, Matt Krop from Risk, Compliance and Threat Management, reviewed all 100 synthetic samples: 66 received an unconditional greenlight, seven were OK with reservations, and 27 did not meet his bar. A “reasonable 100” that has not been through that pass is still silver. Langfuse’s coverage rule is the reason not to pad: 100 diverse items beat 1000 near-duplicates, and a dataset of easy cases scores 95 percent forever. GetMaxim’s 246 is a planning formula for one slice, not a site-wide budget; adjust upward for multi-turn simulations, language variants, and high-risk categories. Where those questions should come from, before you chase any of those counts, is the next constraint.

Where do RAG test questions come from?

RAG test questions come from production traffic first, from domain experts second, and from synthetic generation only to fill coverage the first two have not yet exercised. Langfuse is blunt about the order: real traffic beats invented examples. Start from traces, add CSV imports of cases you already have, and generate synthetically where real data is thin. The highest-value new items are the cases the application just failed. GetMaxim’s sourcing list is the same ladder with names: production logs across segments and difficulty, user-research prompts with consent, SME-authored must-pass scenarios with explicit acceptance criteria, and adversarial safety cases. DataAspirant aims for 20 to 50 drawn from real usage and important document sections, and includes unanswerable and near-miss questions deliberately.

Microsoft’s silver path is how you get to 100 when experts cannot spend two weeks writing 90 pairs. Collect documents, extract location metadata, select chunks, prompt a model to generate question-answer pairs from those chunks, and preserve source document IDs so retrieval success can be scored later. They recommend generating more than one pair per chunk combination to save endpoint cost, and they treat that output as silver until humans review it. In their earnings-call case study, 27 of 100 failed a finance expert’s bar even when the answers were often correct but incomplete: missing dollar amounts, missing a second reason for a change, or quoting landing language instead of the SKU difference asked for. That is why silver is not gold.

Synthetic generation at agent scale, critique loops, and bias audits is its own method: generating synthetic evaluation data. Mining live traffic as a continuous sample, rather than as a one-time seed, belongs under online evaluation. Red Hat’s 23 February 2026 developer article on synthetic data for RAG evaluation covers that synthetic half, not the labelled file itself. An honest set still has to survive leakage, unreviewed silver, and missing unanswerables, or the counts above measure the wrong thing.

How do you keep a golden test set honest?

An honest golden set measures the system; a leaked, duplicated, or unreviewed set flatters it. GetMaxim’s core principles (captured 18 September 2026) are the checklist those guides share: defined scope, representative of production, diverse, decontaminated from training and fine-tuning data, and dynamic. Decontamination, in that guide, means exact and substring matching against known training corpora when available, continuation tests for memorisation, and embedding similarity to cluster near-duplicates and prune redundant items. Langfuse’s trust sentence is the cost of skipping review: an unreviewed pile produces scores nobody acts on. Microsoft’s 27 of 100 failing an expert bar is the measured version of that cost on a real silver set.

Microsoft also names the human failure modes that survive even expert-authored gold. People copy-paste the same answer onto variations of a question that asked what versus how. Tribal knowledge sends the evaluator to a source that is not in the corpus, or is not sufficient. Some questions have no answer. Some answers contain only a link, with no content to use as ground truth. DataAspirant’s forgotten types are the probes that catch a system that always answers: unanswerable questions, and near-misses instead of happy paths. Twenty questions the system obviously handles measure nothing. Statsig’s organisational claim is why this file has to be shared: without a single source of truth, teams relitigate what good looks like on every experiment.

Exact-string match is too strict for grading free-form answers, which is why Microsoft runs Azure AI Studio metrics such as groundedness, relevance, and coherence on the silver file itself, then has a human judge review low-scoring tuples. Bias, agreement, and why two judges disagree on the same pair live on LLM-as-a-Judge for RAG. Before anyone scores a model against the file, validate required fields so a missing source ID cannot silently drop a retrieval case:

import json, sys
from pathlib import Path

REQUIRED = {"id", "question", "reference_answer", "relevant_source_ids", "answer_type"}
ALLOWED_TYPES = {"factual", "multi_doc", "ambiguous", "unanswerable"}

path = Path(sys.argv[1] if len(sys.argv) > 1 else "golden_set.jsonl")
rows, errors, types_seen = [], [], set()
for i, line in enumerate(path.read_text().splitlines(), 1):
    if not line.strip():
        continue
    row = json.loads(line)
    missing = REQUIRED - row.keys()
    if missing:
        errors.append(f"line {i}: missing {sorted(missing)}")
    if row.get("answer_type") not in ALLOWED_TYPES:
        errors.append(f"line {i}: bad answer_type={row.get('answer_type')!r}")
    if row.get("answer_type") == "unanswerable" and row.get("relevant_source_ids"):
        errors.append(f"line {i}: unanswerable should have empty relevant_source_ids")
    types_seen.add(row.get("answer_type"))
    rows.append(row)

if "unanswerable" not in types_seen:
    errors.append("set has zero unanswerable probes: add honesty cases")
if errors:
    print("FAIL"); print("\n".join(errors)); sys.exit(1)
print(f"OK: {len(rows)} cases, types={sorted(t for t in types_seen if t)}")

The validator is a contract on the schema illustration above, not a published industry tool. It fails a file with no unanswerable probes, because DataAspirant and the honesty cases in the JSONL treat that gap as a defect. When the corpus itself changes, even an honest file goes stale, and that is a different failure from leakage.

How do you update a golden set when documents change?

A golden set is maintained the way a test suite is maintained: escaped bugs become cases, duplicates get merged, and obsolete cases get retired on purpose. Langfuse’s freshness warning is the silent decay: a set assembled in January describes January’s traffic; by July, users ask different questions and the product has new features. Treat the dataset as an append-mostly log that tracks production, and date items in metadata so staleness is visible. GetMaxim’s versioning step maps dataset versions to prompt versions and agent workflows, and uses aggregate evals across critical slices as release gates. DataAspirant says treat the file like code: version it, review changes, and grow it every time a real user surfaces a failure you did not anticipate. A golden set that never grows is a test suite for last quarter’s product.

When documents change, re-map stable source IDs after re-ingestion so labels still point at the right sections, and retire cases whose evidence was deleted. Do not leave orphaned relevant_source_ids pointing at a policy that no longer exists. Re-check unanswerables: as the corpus grows, a question that had no answer can become answerable, and an “I don’t know” case that starts passing for the wrong reason is a silent honesty regression. Langfuse’s production-failure intake is the growth rule that keeps the set honest without inventing cases: flagged traces flow into the dataset while the failure is fresh, through an annotation queue or a scheduled sweep of negative feedback. GetMaxim calls the same motion drift management: continuously add real failure cases and update content domains.

Wire the versioned file into CI on regression testing. Langfuse keeps a fast subset of tens to low hundreds for pull-request gates and reserves the full 100-to-1000 set for release branches or nightly runs. If offline scores stay high while users see old answers, the index can be stale even when the golden labels are current; that is stale index, not only a test-set problem. Public leaderboards are a different artefact from this custom file; they live with RAG benchmarks.

A golden test set is the labelled ground RAG metrics stand on: questions you care about, answers a reviewer will defend, and source IDs that still point at the right section after the next re-chunk. Microsoft’s 100, Langfuse’s 20-to-50 then 100-to-1000, GetMaxim’s 246-per-slice planning figure, and DataAspirant’s 20-to-50 are ranges for different jobs, not one target. Production traces first, experts second, silver last, then a human pass that can fail 27 of 100. Version the file, date the items, and add the failure you just shipped, or the scores describe a product you no longer run.