Regression Testing a RAG System in CI

RAG regression testing runs a frozen golden set through the live pipeline on every change and asserts quality metrics stay above thresholds — catching drift before merge. Covers what to assert, what to sample, and how to handle non-determinism.

What is RAG regression testing?

RAG regression testing adapts software regression testing for LLM output: instead of asserting exact strings, you assert that faithfulness, context recall and related scores stay above floors on a curated golden dataset (qaskills, RAG Regression Testing, 2026). Wire that suite into CI and every pull request that touches a prompt, retriever, model or corpus is gated automatically. The failure mode it exists to catch is quality drift — a change that ships with green unit tests and unchanged latency while answer quality quietly falls.

A typical sequence from the 2026 regression guides: a teammate shortens the system prompt and drops the “answer only from context” instruction; faithfulness moves from about 0.94 to about 0.81 on the golden set while users start seeing plausible fabrications (qaskills’ illustrative drift story — calibrate against your own baseline). Other triggers are an embedding upgrade that reshuffles rankings, a re-chunk that moves boundaries, a corpus re-index, a silent provider model update, and dependency bumps that change tokenization (qaskills; Medium / Uttmesh, RAG Testing, 2026). Offline evaluation method is on how to evaluate a RAG system; this page is the CI gate that makes those scores block a merge.

What does a RAG regression suite include?

A working suite has five parts. Miss any one and the gate either cannot decide or cannot explain a failure:

How to build and keep that golden file honest — schema, size ranges, leakage checks — is building a golden test set. This page assumes the file exists and focuses on gating it.

What should you assert in a RAG CI gate?

Assert component metrics that localise the break, not only a single end-to-end correctness score. An end-to-end number tells you “the answer got worse”; faithfulness and context recall tell you whether the generator started fabricating or the retriever stopped fetching evidence (qaskills’ per-component vs end-to-end framing, 2026).

The pattern that holds up in the live CI guides: make faithfulness (groundedness) and context recall hard gates that block merge; treat context precision and answer relevancy as soft gates that warn on the pull request without blocking on modest noise (qaskills, 2026). Split by layer so the bisect is cheap — ContextRelevance down with Groundedness stable points at the retriever; the opposite points at the generator (Future AGI, Evaluate RAG in CI/CD, 2026; Kartik, dev.to, 2026).

Set floors slightly below your current measured baseline so the gate catches real drops instead of aspirational noise (qaskills, 2026). Metric definitions and cross-tool disagreement live on the metrics catalogue and generation metrics; judge bias and agreement rates are on LLM-as-a-judge.

How many cases should a CI regression sample run?

There is no single published magic N for a CI sample — the live guides give ranges tied to cost and signal, and you calibrate:

Composition beats raw count. Cover happy-path intents, multi-hop questions, refusal / unanswerable probes, and the hardest historical production failures (qaskills; Future AGI, 2026). Include expected_chunks or stable source IDs — without them you can score generation but not retrieval, and bisecting a regression takes a day instead of an hour (Kartik, 2026). How to author and version that file is on golden test sets.

How do you wire RAG regression tests into GitHub Actions?

Treat the suite like ordinary pytest: generate answers and retrieval contexts at evaluation time against the code under test, score them, and fail the job when a hard metric breaches its threshold (Confident AI / DeepEval CI guide, 2025; DeepEval docs, Unit Testing in CI/CD). Do not freeze actual outputs in the dataset — that tests a snapshot of yesterday’s pipeline, not the PR.

As of 2026-07-27, PyPI reports deepeval==4.1.4 and pytest==9.1.1. Pin them so the gate does not silently change when the framework moves (the same discipline as the pipeline build pins). A Confident AI cloud key is optional — DeepEval’s docs state an OPENAI_API_KEY (or other judge key) is enough to run locally in CI.

import json
from pathlib import Path

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, ContextualRecallMetric

# Replace with your live RAG call: (answer, list[str] contexts)
from my_rag import run_rag

CASES = json.loads(Path("eval/golden_dataset.json").read_text())["cases"]

faithfulness = FaithfulnessMetric(threshold=0.90)   # hard gate — starting floor from qaskills 2026
recall = ContextualRecallMetric(threshold=0.85)     # hard gate — starting floor from qaskills 2026

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_rag_case(case: dict) -> None:
    answer, contexts = run_rag(case["query"])
    test_case = LLMTestCase(
        input=case["query"],
        actual_output=answer,
        expected_output=case["ground_truth"],
        retrieval_context=contexts,
    )
    assert_test(test_case, [faithfulness, recall])

The workflow below path-filters so docs-only PRs do not burn judge tokens, and exits non-zero when deepeval test run sees a threshold breach:

name: rag-eval-gate

on:
  pull_request:
    paths:
      - "prompts/**"
      - "retriever/**"
      - "rag/**"
      - "eval/**"
      - ".github/workflows/rag-eval.yml"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - name: Install pinned eval deps
        run: pip install deepeval==4.1.4 pytest==9.1.1
      - name: Run RAG regression eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: deepeval test run test_rag_regression.py

Framework choice — DeepEval, Ragas, Phoenix, Braintrust and the rest — is scored on evaluation tools; Ragas-specific metric assumptions are on Ragas. The wiring pattern is the same: pinned scorer, golden inputs, fail the job on breach.

How do you stop RAG CI gates from flaking?

A flaky gate gets muted, and a muted gate is worse than no gate (qaskills, 2026). Both the system under test and the LLM judge are probabilistic, so the same query can yield a slightly different answer and a slightly different score across runs.

Three techniques from the 2026 regression guides, plus DeepEval’s explicit flaky flag:

1. Pin generator temperature to 0 (or near it) in eval even if production runs hotter — you are testing a stable configuration, not sampling creativity (qaskills, 2026).
2. Pin the judge model and set judge temperature to 0. A score from one judge is not comparable to a score from another; changing the judge requires re-baselining the suite (qaskills; AI Engineering Playbook, 2026).
3. Median-of-N on high-variance cases only — run the case a small number of times (guides show N=3), take the median, and fail only if that aggregate is under the floor (qaskills, 2026).

def stable_score(case: dict, measure, runs: int = 3) -> float:
    """Damp judge/generator noise; reserve multi-run for known-flaky cases."""
    scores = []
    for _ in range(runs):
        answer, contexts = run_rag(case["query"])
        scores.append(measure(case["query"], answer, contexts))
    scores.sort()
    return scores[len(scores) // 2]  # median

DeepEval’s docs (Unit Testing in CI/CD) also let you mark a case or metric flaky=True: a failure prints a warning instead of raising, so CI keeps going while the result is still recorded — use that for known threshold-sitters, not as a way to ignore real regressions. Once you have a trailing baseline, upgrade from absolute floors to a delta gate (Welch’s t-test on per-example scores against a 7-day baseline, fail when the drop is significant and above a minimum effect size — Future AGI / Kartik, 2026). Do not invent p-values for your corpus; implement the test against your stored per-example scores. Why judges disagree with humans is covered on LLM-as-a-judge.

How do you version prompts, indexes, and judges for comparable scores?

A CI score means nothing unless prompts, data snapshots, retrieval configuration and the judge are versioned as first-class artifacts so each number is tied to a reproducible system state (AI Engineering Playbook, RAG Evaluation in CI/CD, May 2026). Without that, a +0.10 faithfulness move can be a real win — or a shorter answer format, a different corpus snapshot, or a quiet judge upgrade.

run_id: 2026-07-27T14-02Z
git_sha: abcdef1
prompt:
  version: "4.2.0"
  hash: "a91f3c"
retriever:
  embedding_model: "text-embedding-3-small"
  top_k: 5
  chunk_size: 512
  reranker: "bge-reranker-v2"
corpus:
  index_snapshot: "kb-2026-07-20"
  document_count: 14820
generator:
  model: "gpt-4o"
  temperature: 0
judge:
  model: "gpt-4o-mini"
  temperature: 0
  deepeval: "4.1.4"

When a metric drops, diff the failing run’s manifest against the last passing one — the changed line is the prime suspect (qaskills, 2026). Reconstructing which stage produced a bad answer in a live trace is tracing.

Should every pull request run the full LLM-judge suite?

No. Running the full LLM-judge sweep on every push is the design mistake that prices the gate out of existence (Future AGI, 2026; Kartik, 2026). Split into three tiers that share the same rubric definitions:

• PR-blocking (every relevant push) — hard faithfulness / recall floors plus cheap deterministic checks (citation validity, schema, latency budget) on the 100–200 case CI sample. Target: minutes, not half an hour. Blocks merge.
• Nightly on main — full LLM-judge stack on the wider versioned set (guides cite roughly 15–30 minutes). Blocks promotion to canary.
• Canary / production sample — same rubrics on a slice of live traffic; alert on rolling drift. This is not CI anymore — sampling rates and implicit signals live under online evaluation and monitoring.

Close the loop: every confirmed production failure becomes a new golden case after review, so the same bug cannot regress twice (niteagent, 2026; Kartik, 2026). Keep rubric definitions in the same repo as application code, pinned next to the prompt and the chunker.