Evaluating RAG in Production With Real Users

Online evaluation scores a live RAG system on real user traffic — sampled reference-free judges, thumbs ratings, and implicit behavioural signals — because a golden set cannot cover live phrasing or distribution shift. It complements offline evaluation; it does not replace it.

What is online evaluation in RAG?

Online evaluation is scoring a deployed RAG pipeline on real user queries after ship — with reference-free judges and/or user signals — not on a labelled golden set. Braintrust (What is RAG evaluation?, captured 2026-07-27) defines it as scoring a sample of production queries with the same scorers you use offline, so you see live phrasing, edge cases, and shifting query patterns. Evidently AI’s RAG evaluation guide puts production monitoring on live user queries and tracks faithfulness, completeness and safety without gold answers. In classical IR terms, Pinecone’s offline-evaluation guide (online vs offline metrics) treats online metrics as user-interaction measures (clicks, CTR) and offline metrics as labelled relevance measured before deploy.

This page is how those live signals become evaluation evidence. Dashboards, alert thresholds and the day-to-day ops stack live on monitoring RAG in production.

How is online evaluation different from offline evaluation?

Offline evaluation runs the pipeline on a curated labelled set before you ship; online evaluation scores real traffic after you ship, where you usually have no gold answer for the query that just arrived.

Braintrust is explicit that both are necessary: offline validates changes before deployment; online detects problems that only emerge at scale. Building and keeping the offline half honest is golden test sets; wiring that set into merge gates is regression testing in CI. Live traffic splits that show variants to users are A/B testing RAG changes — different from scoring one shipped system online.

What implicit signals can you use for RAG evaluation?

Implicit signals are user behaviours that indicate the pipeline failed without a thumbs widget — a rephrase of the same intent, an edit of the answer, an escalation to a human, abandoning the session, or retrieving the correct source from another tool. Atlan (RAG Evaluation, 2026) states the rule directly: every time a user edits an answer, follows up to flag an error, or manually retrieves the correct information elsewhere, that is a production evaluation signal. Dextralabs (Production RAG, 2025) logs per-query feedback events and treats spikes in rating drops as anomaly triggers. Braintrust lists thumbs-up / thumbs-down alongside faithfulness and relevancy as continuous monitoring signals.

None of these replace labelled retrieval metrics or generation metrics on a held-out set. To attribute a signal to retrieve vs generate, you need the span — see tracing a RAG pipeline.

How do thumbs-up and thumbs-down ratings work for RAG?

Thumbs (or a one-click “was this helpful?”) are explicit binary ratings stored with the query, the answer, and the retrieved context — sparse labels, but among the highest-value production signals you can collect. ragaboutit (RAG Measurement Framework) recommends embedding a one-click feedback control and storing the rating beside query, response and retrieval context. Dataworkz’s call-center case uses agent thumbs-up to accumulate validated “golden answers.” Dextralabs merges thumbs up/down with answer-accuracy checks as a human feedback loop score.

Limits matter as much as the widget. Unhappy users rate more often than satisfied ones (selection bias). A thumbs-down does not say whether the wrong chunk was retrieved or the generator hallucinated — you still need the trace. A typical thumbs-up rate for RAG is not published as an industry figure; do not invent one. Negative thumbs are the best candidates to promote into golden test sets.

What sampling rate should you use for production evaluation?

A universal sampling rate for online RAG evaluation is not published. Platforms document sampling as a cost-and-coverage trade-off you choose — not a fixed industry percentage.

Pick N from your judge budget and the confidence you need on your baseline, write that rate down in-house, and trend it — do not copy an unsourced “industry %.” Judge cost and bias that drive this trade-off are covered on LLM-as-a-judge; continuous alert wiring (as opposed to sampled scores) is on monitoring.

What is shadow evaluation for RAG?

Shadow evaluation replays logged production queries offline against candidate pipeline configurations — a new embedding model, index, reranker or prompt — without showing those candidate answers to users, then promotes only configs that match or beat the baseline. Adnan Sattar (RAG Engineering Part 5: Monitoring, Evaluation, and Lifecycle Management) describes the loop: log real production queries (with anonymisation where required) → replay offline against candidates → compare retrieval sets, answer-quality scores, latency and cost → promote winners.

Shadow is not the same as live online scoring (users see one shipped system) and not the same as A/B testing (users see variants). Use shadow to de-risk a change on the real query mix; still gate the change with regression tests on the golden set before full rollout.

How do you turn production traffic into an evaluation set?

The production→evaluation flywheel is: instrument traces → score or sample live traffic → promote low-score and thumbs-down cases into the offline golden set → re-run offline evaluation before the next ship.

1. Trace every stage — query, retrieved chunks, prompt, answer, latency (Braintrust @traced pattern; Langfuse online/offline loop).
2. Score a sample online — reference-free faithfulness/relevancy plus thumbs and implicit signals.
3. Promote failures — Braintrust documents converting a production trace into a test case and feeding low-scoring queries into offline datasets; Evidently recommends expanding the test set with real user queries and observed failures once you are in production.
4. Route corrections to the index — Atlan’s production rule: a user correction is not only a log line; it should update the retrieval index when the content was wrong or missing.
5. Re-run offline — the enlarged set becomes the next CI gate on test sets (watch leakage: never tune prompts on the held-out slice).

When does online evaluation miss a failure?

Online evaluation misses failures when feedback is sparse (few thumbs), the sample skips a rare query slice, or the corpus and query distribution drift faster than your flywheel can absorb new cases. Braintrust notes that corpus updates, model updates and evolving query patterns degrade RAG pipelines in ways that are hard to predict from a static golden set alone.

• Sparse thumbs, happy dashboards — enlarge the sample or add implicit signals; do not trust a quiet rating widget.
• Sampled judge flat, users unhappy — the sample missed the failing slice, or the judge disagrees with users; calibrate on LLM-as-a-judge.
• Quality degrades week over week — leave this page for the failure leaf: why RAG gets worse over time, and the ops half on monitoring. Embedding-space mismatch after re-index is embedding drift.