CI/CD and Quality Gates for RAG

A RAG CI/CD quality gate runs offline evaluation on every relevant change and fails the build when retrieval or answer metrics breach a threshold — so a prompt, chunker, index or model tweak cannot merge on green unit tests alone. Evaluation is the pre-ship half; monitoring is the live half. This page is what to gate, how to enforce the fail, and how to keep the gate from becoming theatre.

Why isn’t a normal CI pipeline enough for RAG?

Code unit tests can pass while answer quality drops. RAG behaviour also moves when the corpus, embeddings, prompts, top-k or model change — none of which a pytest on pure functions catches. Dextralabs’ production RAG guide (2025) states the rule bluntly: CI for AI must gate on metrics, not just syntax, and teams that do not version test data waste days chasing regressions that were untracked input changes. Future AGI’s 2026 CI playbook opens on the failure mode: a green check on a tiny mean-above-floor suite is theatre when the dataset never covered the query class that breaks in production. Niteagent’s metrics-to-gates guide (12 June 2026) frames the opposite of a gate — ad-hoc eyeballing with no regression detection. Treat eval as the regression suite for the retrieve→generate path. Metric definitions live at evaluation; the versioned ship unit is at versioning.

What should a RAG quality gate measure?

Gate retrieval and generation separately so a failing PR tells you which layer moved. Confident AI’s DeepEval CI guide and RAGAS-style stacks name the usual surface: contextual precision, contextual recall, faithfulness and answer relevancy — definitions and math stay on the evaluation nodes; this page only consumes the scores. The Green Report’s RAG Triad uses answer relevance, context relevance and groundedness, with example starting thresholds above 0.7 for the relevance metrics and 0.8 for groundedness — their calibration, not a site-wide law. Niteagent’s sample DeepEval thresholds (for example faithfulness 0.85, contextual precision 0.8) are likewise examples in their test file. Future AGI’s split rule is the useful bisect: ContextRelevance drops while Groundedness holds ⇒ retriever; the reverse ⇒ generator. Optional: fail the same suite on a latency SLO (Dextralabs). Deeper metric pages: retrieval metrics and generation metrics.

How do you build and maintain the eval dataset the gate runs on?

The dataset is the gate’s worldview — composition beats raw size. Future AGI argues a 200-example set sampled from production beats 2,000 imagined cases, and cites a PR-blocking sweet spot around 100–200 examples per route (attribute their ranges; do not invent coverage percentages). Niteagent curates a golden set on the order of 100–500 queries in git, appends production failures to a JSONL capture file, and promotes reviewed cases on a weekly cadence. Dextralabs and Towards Data Science both version the eval set with the system under test and grow it from live edge cases. A one-shot launch set goes stale as user behaviour shifts (Niteagent’s explicit pitfall). How to grow sets further is at test sets and synthetic data.

How do you enforce a quality gate in CI?

Run the eval suite in the pipeline and exit non-zero on threshold breach so merge or deploy stops. That fail-closed exit is the mechanism every working guide shares; the runner varies.

• DeepEval + pytest — assert_test with per-metric thresholds; the test case fails unless every metric passes (Confident AI).
• CLI fail flag — Niteagent’s GitHub Actions example runs DeepEval with --fail-on-threshold-breach so a breach blocks the PR.
• RAGAS in CircleCI — CircleCI’s blog wires a RAGAS evaluator job into .circleci/config.yml.
• TruLens triad + exit code — The Green Report aggregates RAG Triad scores and returns a non-zero process exit when any metric misses its floor.
• Reference architectures — Google Cloud’s Harness CI/CD for RAG apps diagram is a staged wiring sketch, not a required vendor.

The neutral, scored comparison of eval platforms lives at evaluation tools — this page names the fail-closed pattern, it does not crown a winner.

How do you keep the gate cheap enough that nobody turns it off?

Future AGI’s CI eval triangle is the design constraint: cheap, fast and statistically meaningful — pick any two and the gate becomes theatre or gets disabled by quarter’s end. Their working answer is trigger tiers: PR-blocking jobs with cheap or deterministic rubrics on a path-scoped subset; a nightly full LLM-judge sweep on main; canary or sampled live scoring with the same rubrics. Their statistical gate pairs an absolute floor with a delta against a rolling baseline (Welch’s t-test framing in their playbook — attribute the method; do not invent p-values as site facts). Cache judge verdicts; shard by route so a legal-RAG PR does not rerun every suite. Bridge CI and prod so the live half at monitoring scores with the same rubric definitions.

How do you wire CI quality gates into a RAG pipeline?

Keep thresholds.yaml and the golden dataset in the same repo as the chunker, prompt and retriever (Niteagent’s layout). Run path-scoped eval on PRs that touch those paths; block merge on breach; promote with the same ship-unit version ids as versioning; feed monitoring failures back into the dataset; never let a deploy skip the gate because “it was only a prompt tweak.” Serving shapes sit at deployment; where the hooks go in a greenfield build is build a pipeline.