Building a Golden Test Set for RAG

A golden test set for RAG is a versioned file of real questions paired with known-good answers and the context those answers should come from — the labelled ground you score retrieval and generation against. Without it, evaluation metrics are vibes (Rajesh Kartha, 2025). This page covers how many questions you need, where they come from, and how to keep the set honest as the corpus changes.

What is a golden test set for RAG?

A golden test set is the labelled benchmark your eval harness scores against: questions you care about, reference answers you trust, and the source passages that should be retrieved. The metrics are easy once that file exists; building it is the actual work (2muchcoffee, How to Actually Evaluate Your RAG, 2026). In the Ragas-style four-input frame, the set supplies the ground truth side — question and ideal answer — while a live run supplies retrieved context and the system answer (Faktion, RAG Output Validation Part 2). How to run the scores is on how to evaluate a RAG system; what each score means is on the metrics catalogue.

What goes in each RAG test case?

Each case needs three things at minimum: the question, a reference answer, and stable IDs for the sources that should be retrieved (Rajesh Kartha’s three-part shape; Jaconir’s question–context–answer triplet). Keep it a flat JSONL file — one object per line (ai-tldr, Build a RAG Evaluation Dataset).

{"id":"q-014","question":"How long do I have to return a physical product?","reference_answer":"Physical items can be returned within 30 days of purchase.","relevant_source_ids":["refunds-policy#returns-window"],"answer_type":"factual","source":"real_support_ticket"}
{"id":"q-031","question":"Can I return a digital download after opening it?","reference_answer":"Digital downloads are not returnable once the download link has been used.","relevant_source_ids":["refunds-policy#digital"],"answer_type":"factual","source":"sme_authored"}
{"id":"q-088","question":"What is our store's policy on time travel refunds?","reference_answer":"I don't know — this is not covered in the corpus.","relevant_source_ids":[],"answer_type":"unanswerable","source":"honesty_probe"}

Use relevant_source_ids that name a document and section, not a transient chunk row number — if you re-chunk the corpus, row IDs break and the golden set lies (ai-tldr). Retrieval metrics need those labels; see measuring retrieval.

How many questions do you need in a RAG test set?

There is no single published magic number. What the live guides actually recommend, as of the 2026-07-27 teardown:

Diversity matters more than raw count (ai-tldr). Cover document slices, question types (factual, multi-doc, ambiguous), and unanswerable probes — not 500 paraphrases of your homepage FAQ.

Where do RAG test questions come from?

Three sources, in rough order of value:

1. Real user queries — support tickets, search logs, sales questions (ai-tldr ranks these first).
2. SME hand-written gold — sit with someone who knows the corpus; write the cases that must not be wrong (Rajesh: start here for high-stakes).
3. Silver synthetic → gold — generate candidates from documents with an LLM, then humans edit them into gold (Microsoft “path to a golden dataset,” May 2024; GetMaxim silver→gold step).

Synthetic generation at scale — agents, critique loops, bias checks — is its own page: generating synthetic evaluation data. Mining live traffic for drift belongs under online evaluation.

How do you keep a golden test set honest?

An honest set measures the system; a leaked or stale set flatters it. The single most damaging failure is test-set leakage — letting evaluation data influence the system under test so scores climb while production does not (ai-tldr).

Exact-string match is too strict for grading free-form answers; the modern default is an LLM-as-judge calibrated against a few human-scored examples (ai-tldr) — bias and agreement rates are on LLM-as-a-judge. Before you trust a file, validate required fields:

import json, sys
from pathlib import Path

REQUIRED = {"id", "question", "reference_answer", "relevant_source_ids", "answer_type"}
ALLOWED_TYPES = {"factual", "multi_doc", "ambiguous", "unanswerable"}

path = Path(sys.argv[1] if len(sys.argv) > 1 else "golden_set.jsonl")
rows, errors, types_seen = [], [], set()
for i, line in enumerate(path.read_text().splitlines(), 1):
    if not line.strip():
        continue
    row = json.loads(line)
    missing = REQUIRED - row.keys()
    if missing:
        errors.append(f"line {i}: missing {sorted(missing)}")
    if row.get("answer_type") not in ALLOWED_TYPES:
        errors.append(f"line {i}: bad answer_type={row.get('answer_type')!r}")
    if row.get("answer_type") == "unanswerable" and row.get("relevant_source_ids"):
        errors.append(f"line {i}: unanswerable should have empty relevant_source_ids")
    types_seen.add(row.get("answer_type"))
    rows.append(row)

if "unanswerable" not in types_seen:
    errors.append("set has zero unanswerable probes — add honesty cases")
if errors:
    print("FAIL"); print("n".join(errors)); sys.exit(1)
print(f"OK — {len(rows)} cases, types={sorted(t for t in types_seen if t)}")

How do you update a golden set when documents change?

Treat the golden set as a test suite in version control next to the code it grades (2muchcoffee). When documents change:

• Re-map stable source IDs after re-ingestion so labels still point at the right sections.
• Retire cases whose evidence was deleted; do not leave orphaned relevant_source_ids.
• Rotate in production failures so the set describes the system you run now, not the one you shipped months ago (2muchcoffee).
• Re-check unanswerables — as the corpus grows, a question that had no answer can quietly become answerable, and an “I don’t know” test that starts passing for the wrong reason is a silent honesty regression (2muchcoffee).

GetMaxim’s golden-dataset guide frames this as versioning, evolution, and release gates. Wire the file into CI on regression testing. If offline scores stay high while users see old answers, you also have a stale index problem — not only a test-set problem.