Monitoring RAG in Production

Monitoring is watching a live RAG system on real traffic — retrieval quality, answer quality, latency, cost and drift — so you catch a regression before your users do. It is not the same thing as evaluation: evaluation scores a test set before you ship; monitoring watches production, where you have no ground-truth answers and every input is one your test set never saw. This page is what to watch, how to watch it without labels, and how to wire it in.

How is monitoring different from evaluation?

They run at different times against different data, and conflating them is why teams get surprised in production. Evaluation runs before you ship, on a labelled test set, and answers “is this change good?”. Monitoring runs after, on live traffic you have no labels for, and answers “is the system still good right now?”. The consequence is the whole difficulty of this page: online you cannot compute faithfulness against a gold answer, because there isn’t one — you infer quality from signals instead. Both are needed. Evaluation catches regressions before merge; monitoring catches drift and the real-world inputs a test set never contained. Offline evaluation is covered at evaluation; this page is the live half.

What should you monitor in a production RAG system?

Five signal groups. Each is worth an alert, because each fails independently:

• Retrieval quality — are the right chunks coming back? Track retrieval scores and hit-rate on a sampled set of queries. Alert: hit-rate drops below baseline.
• Generation quality — is the answer faithful and relevant? Approximate with an LLM-judge on a sample plus user feedback. Alert: judge score trends down.
• Latency — the per-stage budget, retrieve versus generate. Alert: p95 crosses your threshold. The stage budget is at latency.
• Cost — tokens, embedding calls and vector-DB spend per query. Alert: cost per query climbs. Deeper at RAG cost.
• Drift — is quality sliding as the corpus and the queries shift away from what you built for? Alert: retrieval or judge scores trend down over weeks. This is drift, watched live.

How do you monitor quality without ground truth?

In production you don’t know the right answer, so you approximate quality from signals you do have. Four proxies, strongest to weakest:

• Tracing — record every retrieval and generation so a bad answer can be reconstructed and attributed to a stage. It underpins everything else; see observability.
• Reference-free LLM-as-judge on a sampled slice — faithfulness of an answer to its retrieved context needs no gold answer, only the context that was used. How this works is at LLM-as-a-judge.
• Implicit user signals — thumbs, a follow-up rephrase, an escalation to a human. Noisy, but real and free.
• Retrieval scores — a cheap leading indicator that something upstream shifted.

Each proxy is noisy on its own, so sample and trend rather than trust a single online score.

What tools do RAG monitoring and observability?

Three categories, and this site sells none of them:

• Eval-native platforms with online modes — Ragas, Phoenix, Braintrust, Evidently — score quality on live samples.
• LLM-observability and tracing — LangSmith, Arize, Datadog LLM — trace requests and surface latency and cost.
• DIY on general observability — OpenTelemetry plus your existing metrics stack, when you want to own it.

Pick on tracing depth, online-scoring support, cost, and open-source versus hosted. The neutral, scored comparison of these tools lives at evaluation tools — this page names the category, it doesn’t crown a winner.

How do you wire monitoring into the pipeline?

Instrument once, at the seams. Emit a trace per request spanning retrieve and generate, with metrics on each stage. Sample a slice for LLM-judge scoring asynchronously, so scoring never adds latency to the live answer. Set alerts on the leading indicators — retrieval hit-rate drop, latency p95, cost per query, judge-score trend. And gate deploys in CI on the offline eval, so a regression can’t ship in the first place. Monitoring and the CI eval gate are the two halves of keeping RAG good: a pre-ship gate and a live watch. Where the hooks go is shown in build a pipeline.