RAG Observability and Tracing

RAG observability is tracing every stage of a retrieve→generate request — query rewrite, embed, vector search, rerank, generate — as structured spans so a bad answer is reconstructable. Monitoring watches aggregates and fires alerts; observability answers which stage failed on this request. This page is the span tree, what each span must carry, and how you debug from it.

How is observability different from monitoring?

They answer different questions on the same live system. Monitoring asks “is the system healthy right now?” with aggregates — retrieval hit-rate, latency p95, cost per query, judge-score trend — and fires alerts when a signal moves. Observability asks “why did this answer go wrong?” by reconstructing one request as a span tree. You need both. Monitoring catches the regression; observability attributes it to a stage. The aggregate signal list and the alert that catches each live at monitoring; this page is the reconstructable half.

How is observability different from evaluation?

Evaluation scores a labelled test set before you ship and tells you whether a change is good. Observability records live requests after you ship so a failure is debuggable. The bridge is an online score — faithfulness of the answer to its retrieved context — attached to the trace as a span event, sampled asynchronously so it does not add latency to the live answer. That scoring method is LLM-as-a-judge; the pre-ship test-set method is evaluation. Observability does not replace either.

What does a RAG trace look like?

One user query is one trace. Each pipeline stage is one span. The minimum set, matching the anatomy FutureAGI, OneUptime and Braintrust all describe in their 2025–2026 guides (captured 28 July 2026):

• Query rewrite — original query and the rewritten form sent to retrieval.
• Embedding — the query vector and which embedding model produced it.
• Retrieve — the chunks returned, with scores and document versions.
• Rerank — before-order and after-order, when a reranker is in the path.
• Generate — the model call, tokens and citations back to chunk ids.
• Judge (optional, async) — a faithfulness score attached after the answer is delivered.

When the retrieve span shows the wrong chunk, the failure mode is wrong chunk.

What should each RAG span capture?

Enough metadata to replay the failure without guessing. The must-capture set, drawn from FutureAGI’s span attributes and Respan’s span schema (captured 28 July 2026):

• Retrieve — chunk_id, similarity score, doc_version, top_k, and retriever strategy (vector, BM25 or hybrid).
• Embed — embedding model id and version (a silent model swap regenerates a different vector space).
• Rerank — input order and output order, so you can see whether the reranker helped.
• Generate — model id, token usage, and citations mapped back to chunk ids.
• Judge — faithfulness (and related) score, written asynchronously onto the same trace.

OpenTelemetry-based instrumenters in 2026 cover retrievers against Weaviate, Pinecone, Qdrant, Chroma and Milvus among others — capture the same attributes regardless of which store you run. Redact PII from chunk text at the collector before storage; retrieval spans carry content the same way LLM message spans do.

How do you debug a bad answer from a trace?

Open the failing trace and walk the spans until the first one that is wrong. That first-wrong-span rule is the whole method:

• Retrieve returned the wrong chunk — fix the index, chunking or query. See wrong chunk.
• Retrieve was right but the reranker reordered badly — fix or remove the reranker.
• Retrieve and rerank were right but the answer is unfaithful — fix the prompt or the generator; the context was fine.
• One span’s latency spiked — that stage owns the budget problem; see latency.

When quality slides over weeks rather than on one request, the signal is drift — still caught first by a trend on traced scores, then diagnosed here.

What tools do RAG observability?

Three categories, and this site sells none of them:

• RAG-native platforms — Langfuse, Phoenix, Braintrust, Galileo and peers — retrieval-aware UIs that show chunk lists on the retrieve span and attach online scores.
• OpenTelemetry DIY — OTel GenAI semantic conventions plus your own backend, when you want to own the store.
• APM and LLM add-ons — Datadog LLM, LangSmith — broader application tracing with lighter chunk-level views.

Pick on retrieval-span depth, online-scoring support, open-source versus hosted, and cost. The neutral scored comparison lives at evaluation tools — this page names the category, it does not crown a winner.