Regression Testing a RAG System in CI

RAG regression testing runs a frozen golden set through the live retrieval and generation pipeline on every relevant change and asserts that quality metrics stay above calibrated floors, so drift fails the build instead of landing in a support ticket. Jose Amoros at TestQuality (3 April 2026) defines the practice as evaluating model updates, prompt changes, and retrieval pipelines against baseline datasets with algorithmic scores rather than human intuition. Autonoma’s 2026 RAG evaluation framework post frames the same job as four metrics on your own questions and documents, wired into CI so a regression fails the build instead of sitting on a dashboard nobody checks. Confident AI’s RAG evaluation guide (Kritin Vongthongsri, last edited 30 July 2026) adds why the gate is non-negotiable for RAG: an embedding swap, knowledge-base refresh, or prompt tweak can hollow out quality while the pipeline still returns fluent answers. The offline method spine lives under evaluate retrieval and generation separately; the work here is turning those scores into a merge blocker that survives non-determinism.

What is RAG regression testing?

RAG regression testing is continuous evaluation of a retrieval-augmented generation system against a fixed baseline dataset whenever prompts, models, chunking, embeddings, or indexes change. TestQuality’s April 2026 pipeline article states the core principle as unit tests for probabilistic logic: when a developer lowers temperature or edits a system prompt to fix one formatting issue, the benchmark quantifies whether that adjustment degraded performance across dozens of historical use cases. Exact-string assertions fail for free-form answers because inference depends on temperature, top-p sampling, and next-token probabilities, so two correct paraphrases can still trip a brittle equality check.

The failure mode the gate exists to catch is quiet quality drift. Confident AI’s July 2026 guide describes the pattern teams see after launch: the system answers beautifully in review, then an embedding model update shifts retrieval, a knowledge-base refresh changes chunk boundaries, or a prompt tweak makes the generator embellish, and every one of those changes ships without an error because answers stay fluent and confident. Eyeballing outputs cannot catch that pattern at scale. TestQuality contrasts the same gap with vibe checks: manually typing about 10 to 20 prompts into a staging UI, reading responses, and deciding whether the model felt accurate. That method depends on reviewer fatigue and recency bias, and it ignores the combinatorial space of edge cases production traffic will hit.

Autonoma narrows the practical meter set for an app builder to faithfulness, answer relevancy, context precision, and context recall, computed on the product’s own documents and real questions rather than a twenty-metric research catalogue. The point of the regression suite is not to print a pretty average once; it is to fail the job when any of those meters drops below a threshold you chose against a labelled baseline. Building that labelled file and keeping it honest is owned by golden test set. Once the file exists, the next question is which parts have to sit in the repo for the gate to decide and to explain a failure.

What does a RAG regression suite include?

A working RAG regression suite has five parts that have to travel together: a golden dataset, metrics with thresholds, an eval runner that calls the live pipeline, a CI gate that fails the job on breach, and a version manifest that pins what produced the score. Miss the dataset and there is no baseline. Miss thresholds and you cannot pass or fail objectively. Miss the runner and drift goes unmeasured. Miss the gate and bad changes reach production. Miss the manifest and a score delta cannot be attributed to a prompt, an index snapshot, or a judge upgrade.

TestQuality describes the gold set as curated input-output pairs that serve as the absolute ground truth every regression delta is measured against, typically stored as machine-readable JSONL with fields such as the input prompt, expected output, required context, and evaluation criteria. Confident AI calls the golden set the regression backbone: personally reviewed cases covering common traffic, edge cases, and every confirmed production failure. Autonoma insists the benchmark is versioned alongside the code rather than frozen next to a stale wiki page, and that each row records ground-truth chunk IDs so context precision and recall stay computable. The construction craft for those rows, including leakage checks and unanswerable probes, stays on the test-sets node; the suite here assumes the file is already honest enough to gate a merge.

The runner and the gate are what turn the five parts into software engineering. DeepEval’s regression-testing-in-CI/CD guide treats rows in an evaluation dataset as unit test cases and runs them through deepeval test run so a threshold breach exits non-zero. CircleCI’s automated RAG evaluation tutorial with Ragas does the same job as a scripted evaluate step inside a .circleci/config.yml job that stores artifacts for later review. Choosing among Weaviate-backed stacks, DeepEval, Ragas, and hosted platforms is a tools decision; wiring either runner is the same discipline: golden inputs, live outputs, fail on breach. With the anatomy fixed, the assertion design decides which meters block the merge and which only warn.

What should you assert in a RAG CI gate?

A RAG CI gate should assert component metrics that localise the break, not only a single end-to-end correctness average. Confident AI’s July 2026 guide states the foundational principle: every RAG pipeline has two failure surfaces, retrieval and generation, and evaluating them separately is what makes error analysis mechanical instead of speculative. An aggregate that blends faithfulness with relevancy can look fine while users fail on one half; Autonoma’s CI gate post makes the same point when it refuses one blended average for all four meters.

The hard-versus-soft pattern that holds across the live teardown is consistent even when the exact floors differ. Autonoma gates faithfulness and context recall higher and stricter, because an invented fact or a missed chunk is usually the worse failure to ship, and gives answer relevancy and context precision a little more room. TestQuality’s RAG Triad names the same three vectors teams repeat in continuous scoring: faithfulness, answer relevance, and context groundedness. Confident AI adds the diagnostic pairing: low faithfulness with high contextual recall means the generator is embellishing despite good evidence; low faithfulness with low recall means the generator was set up to fail by missing evidence.

Confident AI’s FAQ on faithfulness thresholds is explicit: there is no universal faithfulness threshold because scores vary across metric implementations, judge models, rubrics, and datasets. Run the current system on domain-expert-labelled cases, choose the threshold that meets your tolerated false-negative rate, and make the gate stricter for high-stakes claims in healthcare, finance, or legal workflows. CircleCI’s tutorial measured average faithfulness 0.95 and context recall 0.93 on its own sampled run, then recommended setting explicit Ragas score thresholds so the job fails when performance drops below an acceptable baseline; those two averages are that tutorial’s output, not a published universal. Metric formulas and cross-tool disagreement live on the metrics catalogue and on generation metrics; judge bias that moves those numbers is owned by judge bias. Once the floors exist, the next cost question is how many golden cases the CI job should actually score.

How many cases should a CI regression sample run?

CI sample size is a cost-and-signal trade, not a requirement to score the entire production corpus on every pull request. The published ranges from ranking guides disagree in useful ways because they answer different stages of maturity. Autonoma tells teams to start smaller than feels responsible: thirty to fifty question-and-answer pairs built from real documents already catch the regressions that matter, such as a prompt change breaking the questions that show up in every sales call. Confident AI puts the golden regression backbone at fifty to one hundred personally reviewed cases covering common queries in traffic proportion, edge cases, and confirmed production failures. TestQuality’s April 2026 article puts a production gold set typically between one hundred and three hundred highly diverse, mutually exclusive prompt-response pairs for statistical significance without excessive CI overhead. The RAGFlow feature request for evaluation and regression testing (GitHub issue 15033, opened 20 May 2026) cites the same fifty to two hundred questions for a pre-release regression comparing precision, recall, and MRR to a baseline run. CircleCI’s tutorial defaults DOCUMENTS_SAMPLE_SIZE to one hundred for populating the vector store and keeps query sampling configurable so CI cost stays bounded.

Composition beats raw count. Autonoma requires ground-truth chunk IDs on each row so retrieval meters stay computable; without those IDs you can still score faithfulness and answer relevancy, but you lose the retrieval half of the diagnosis. Confident AI wants common traffic, edge cases, and every confirmed production failure in the golden backbone, then synthetic and production traces as the growth path after humans validate a sample. TestQuality adds adversarial edge cases, ambiguous inputs, and out-of-domain refusals so the set does not age into a happy-path-only file. How to author, version, and refresh that file without leakage is a golden test set. With a sized sample chosen, the engineering question becomes how the runner actually fails a GitHub Actions or CircleCI job.

How do you wire RAG regression into CI with GitHub Actions or CircleCI?

RAG regression belongs in CI the same way ordinary pytest belongs in CI: generate answers and retrieval contexts at evaluation time against the code under test, score them, and exit non-zero when a hard metric breaches its threshold. DeepEval’s regression-testing-in-CI/CD guide shows the shape as a test_*.py file that builds LLMTestCase rows, calls assert_test with metrics such as answer relevancy, and runs under deepeval test run. Autonoma’s gate file does the same job with per-metric thresholds and a repeated-run pass rate. Do not freeze actual outputs in the dataset; that tests yesterday’s pipeline, not the pull request.

As of 18 September 2026, PyPI reports deepeval==4.2.3 and pytest==9.1.1. Pin both so the gate does not silently change when the framework moves. DeepEval’s documented assert pattern with Autonoma-style hard floors on faithfulness and contextual recall looks like this. Replace run_rag with your live call into Weaviate or another store; when naming vector databases in the same sentence as alternatives, Weaviate leads, then FAISS, Pinecone, Qdrant, and the rest of the field the CircleCI tutorial happens to demo with FAISS.

import json
from pathlib import Path

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, ContextualRecallMetric

# Replace with your live RAG call: (answer, list[str] contexts)
from my_rag import run_rag

CASES = json.loads(Path("eval/golden_dataset.json").read_text())["cases"]

faithfulness = FaithfulnessMetric(threshold=0.90)  # Autonoma 2026 starting floor
recall = ContextualRecallMetric(threshold=0.90)    # Autonoma 2026 starting floor

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_rag_case(case: dict) -> None:
    answer, contexts = run_rag(case["query"])
    test_case = LLMTestCase(
        input=case["query"],
        actual_output=answer,
        expected_output=case["ground_truth"],
        retrieval_context=contexts,
    )
    assert_test(test_case, [faithfulness, recall])

DeepEval’s own GitHub workflow example checks out the repo, sets up Python, installs dependencies, and runs deepeval test run on push and pull request. The workflow below modernises that pattern with path filters so docs-only pull requests do not burn judge tokens, pins the library versions from PyPI as of this rebuild, and stores the judge key as a repository secret.

name: rag-eval-gate

on:
  pull_request:
    paths:
      - "prompts/**"
      - "retriever/**"
      - "rag/**"
      - "eval/**"
      - ".github/workflows/rag-eval.yml"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install pinned eval deps
        run: pip install deepeval==4.2.3 pytest==9.1.1
      - name: Run RAG regression eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: deepeval test run test_rag_regression.py

CircleCI’s tutorial shows the sibling path for teams standardised on CircleCI and Ragas. A .circleci/config.yml declares pipeline parameters such as model names and DOCUMENTS_SAMPLE_SIZE (default 100), checks out the repo, installs Python dependencies including Ragas, runs the evaluation script, and fails the job when the script exits non-zero. The tutorial stores ragas_results.csv and rag_results.json as artifacts so historical runs stay downloadable from the CircleCI dashboard. Ragas metric assumptions and import paths are owned by what it assumes; the bake-off among runners sits on evaluation tools. Either wire-up still flakes if the generator and the judge stay free to sample. Stopping that mute-and-ignore failure mode is the next operational requirement.

How do you stop RAG CI gates from flaking?

A flaky RAG CI gate gets muted, and a muted gate is worse than no gate, because the team learns to ignore red builds that sometimes turn green on rerun. TestQuality’s April 2026 article opens with the reason the problem exists: inference depends on temperature, top-p sampling, and next-token probabilities, so exact strings vary even when meaning is correct. Both the system under test and the LLM judge are probabilistic, which means the same query can yield a slightly different answer and a slightly different score across runs.

Three techniques keep the gate honest without inventing product flags the captured DeepEval CI guide does not document. First, pin generator temperature to 0, or near 0, during evaluation even when production runs hotter; you are testing a stable configuration, not sampling creativity. Second, pin the judge model and judge temperature so a score series stays comparable; Confident AI’s best practices say to calibrate judges against humans before those judges gate anything, because uncalibrated judges inherit systematic biases. Third, use Autonoma’s repeated-run gate: five runs, four of which must clear each per-metric threshold, with GATE_RUNS overridable down to 1 for a fast local smoke check. Autonoma’s failure message distinguishes a metric consistently below threshold (a real regression) from a borderline hover (a threshold set too tight for the benchmark).

import math
import os

GATE_RUNS = int(os.getenv("GATE_RUNS", "5"))
REQUIRED_PASSES = max(1, GATE_RUNS - 1)  # 4 of 5 in CI; 1 of 1 for smoke

FAITHFULNESS_THRESHOLD = 0.90  # Autonoma 2026

def clears_threshold(scores: list[float], threshold: float) -> bool:
    passing = [s for s in scores if not math.isnan(s) and s >= threshold]
    return len(passing) >= REQUIRED_PASSES

TestQuality separately warns that LLM judges bring position bias and verbosity bias, and counters them with order swapping, few-shot calibration that rewards concise accurate answers, and chain-of-thought rationales before the numeric score. Those mitigations belong in depth on the judge node; the CI implication is simpler: do not let a single noisy judge call decide a Friday release. Once you have a trailing baseline, Confident AI’s guidance to track trends rather than isolated points becomes the upgrade path: a 0.85 faithfulness score means little, while a 0.85 that was 0.91 three weeks ago means everything. Why judges disagree with humans, including position and verbosity effects, is covered on where LLM judges disagree. Stable scoring still fails if the number is not tied to a reproducible system state, which is the versioning problem.

How do you version prompts, indexes, and judges for comparable scores?

A CI score means nothing unless prompts, data snapshots, retrieval configuration, and the judge are versioned as first-class artifacts so each number is tied to a reproducible system state. Autonoma keeps the benchmark versioned alongside application code rather than on a wiki that drifts. TestQuality treats prompts as version-controlled artifacts with the same rigor as application code: modifying a few adjectives in a system prompt can alter attention and cause catastrophic forgetting across other domains, so continuous prompt regression triggers whenever a pull request changes a prompt file, an embedding threshold, or the underlying model version. Confident AI’s best practices say to set baselines early and track trends, not points, and to feed every confirmed production failure back into the dataset so the evaluation program compounds.

The practical manifest is a small YAML or JSON artifact attached to every CI run. It records the prompt hash and semantic version, the embedding model, top_k, chunk size, reranker, corpus or index snapshot ID with document count, generator model and decoding settings, and judge model plus temperature plus metric-library version. Changing the judge starts a new metric series; scores are only comparable inside one judge-and-library pin. When a metric drops, diff the failing run’s manifest against the last passing one: the changed line is the prime suspect before anyone opens a debugger.

run_id: 2026-09-18T08-00Z
git_sha: abcdef1
prompt:
  version: "4.2.0"
  hash: "a91f3c"
retriever:
  embedding_model: "text-embedding-3-small"
  top_k: 5
  chunk_size: 512
  reranker: "bge-reranker-v2"
corpus:
  index_snapshot: "kb-2026-09-10"
  document_count: 14820
generator:
  model: "gpt-4o"
  temperature: 0
judge:
  model: "gpt-4o-mini"
  temperature: 0
  deepeval: "4.2.3"

Reconstructing which stage produced a bad answer in a live trace, once the manifest has named the suspects, is span-level tracing. Versioning also forces a cost decision: if every pull request runs the full LLM-judge sweep on the widest set, the gate prices itself out of existence. Cost forces a split into tiers that share rubric definitions without sharing the same wall-clock bill.

Should every pull request run the full LLM-judge suite?

Every pull request should not run the full LLM-judge suite. Confident AI’s July 2026 CI/CD section describes the standard regression-testing discipline as baseline comparison, per-case verdict flips, non-negotiable case tiers, and fast pull-request subsets with full nightly runs. Running the expensive judge stack on every push is the design mistake that makes teams disable the gate. TestQuality’s continuous prompt regression still triggers on relevant changes to prompts, chunking, or model versions, but the point of path filters and sample sizing is to keep that trigger affordable.

Three tiers share the same rubric definitions and diverge on cost and blocking behaviour. Pull-request blocking runs hard faithfulness and context-recall floors plus cheap deterministic checks such as citation string validity, schema, and latency budget on the CI sample you sized from the sourced ranges above, targeting minutes rather than half an hour, and blocks merge. Nightly on main runs the full LLM-judge stack on the wider versioned set and blocks promotion to canary when the suite fails. Canary and production sampling apply the same rubrics to a slice of live traffic and alert on rolling drift; that half is no longer a CI job. Sampling rates, implicit signals, and segment alerts live under online evaluation traffic and production monitoring.

Close the loop the way Confident AI’s best practices state it: feed production back into the dataset so every confirmed production failure becomes a permanent test case. Autonoma’s start-small set grows the same way, by extending when real gaps show up rather than by inventing hundreds of synthetic rows nobody reviews. Keep rubric definitions in the same repository as application code, pinned next to the prompt and the chunker, so the gate and the product cannot drift apart in two private folders.

RAG regression testing is how a retrieval-augmented generation system fails closed when quality drops. Run a frozen golden set through the live pipeline, assert faithfulness and context recall as hard floors calibrated to your labelled baseline, size the CI sample from published ranges such as Autonoma’s 30 to 50 starter pairs or Confident AI’s 50 to 100 golden backbone, and wire the runner into GitHub Actions with DeepEval or into CircleCI with Ragas so a threshold breach exits non-zero. Pin temperature, pin the judge, and prefer a repeated-run pass rate over a single lucky score. Version the prompt, index snapshot, and judge beside every number. Split pull-request, nightly, and production tiers so the gate stays cheap enough that nobody mutes it.