Tracing a RAG Pipeline: Seeing Which Stage Failed

Tracing a RAG pipeline records one request as a tree of spans — embed the query, retrieve chunks, assemble context, generate the answer — so a bad response is attributable to a stage instead of a black box. This page is the instrumentation and the four trace signatures that identify almost every failure. Metric definitions live under evaluation; the hour-scale debug procedure under finding which stage broke; live dashboards under monitoring.

How is RAG tracing different from evaluation?

Evaluation scores whether the system is good; tracing records what one request did. Evaluation runs a labelled set and returns metrics such as context recall or faithfulness. Tracing stores the span tree for a single query — the retrieved chunk ids, the assembled prompt, the model output — so you can see which stage produced a bad answer. A score without a span tree cannot name the failing stage; a beautiful trace without scores cannot say the answer was wrong. You need both. Live traffic dashboards, alerts and sampling rates are the production half — covered under monitoring and production observability; turning production traffic into an eval set is online evaluation.

What does a RAG trace contain?

A RAG trace is one root span per user query and a child span per pipeline stage. The minimum useful set, carried across the top-ranking results how-tos (OneUptime, Feb 2026; Future AGI, 2026; Respan, May 2026), is:

• embed_query — embedding model id and version used to vectorise the question.
• retrieve — strategy (dense / BM25 / hybrid), top-k, and for each hit: chunk_id, similarity score, doc or index version. Chunks are attributes on this span, not child spans (Future AGI, 2026 — span-per-chunk explodes trace size).
• assemble — which chunk ids entered the prompt, and whether truncation dropped evidence.
• generate — model id, prompt version or hash, token usage, and the answer text (redacted if needed).

Optional children — query rewrite, rerank, grounding/judge — attach when those stages exist. The stages themselves are the same six you build in the pipeline tutorial; tracing is how you watch them at query-time.

Which four traces identify almost every failure?

Four readable signatures on the span tree identify almost every RAG failure. Maxim AI’s complete guide to tracing and evaluating RAG pipelines (live teardown, 2026) names the same four failure areas in the abstract; the table below is what those areas look like when you open a trace — so you stop guessing which stage to fix.

If you only know “something is wrong” and need the ordered measurement procedure, use how to find which stage broke. The failure taxonomy maps every symptom to a stage once the signature is clear.

How do you instrument a RAG pipeline with OpenTelemetry?

You instrument a RAG pipeline with OpenTelemetry by creating one tracer and a child span per stage, then exporting via OTLP to whatever backend already stores your traces. Pin the SDK versions so the tutorial still runs later — PyPI current as of July 2026:

pip install opentelemetry-api==1.44.0 
            opentelemetry-sdk==1.44.0 
            opentelemetry-exporter-otlp-proto-http==1.44.0

Configure a tracer provider once at process start. The exporter endpoint is whatever your collector accepts — set it from the environment, never hard-code a vendor URL into application code:

import os
from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

resource = Resource.create({
    "service.name": "rag-pipeline",
    "service.version": "1.0.0",
})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint=os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"],
        )
    )
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("rag.pipeline", "1.0.0")

Each stage opens a child span and records the attributes the four signatures need. Auto-instrumentation libraries such as OpenInference and OpenLLMetry (named across Future AGI and the OpenTelemetry LLM observability post, 2024) cover many LLM and vector-store clients — including Weaviate, Pinecone, Qdrant and Milvus — but retrieve and assemble still need RAG-specific attributes you set yourself. As of 2026 the OpenTelemetry GenAI semantic conventions cover LLM calls better than retrievers; treat retrieve/assemble as custom spans until the conventions catch up (Dev.to / erythix, 2026).

def answer(self, question: str, top_k: int = 4) -> str:
    with tracer.start_as_current_span("rag.query") as root:
        root.set_attribute("rag.query.text", question)

        with tracer.start_as_current_span("rag.embed_query") as span:
            span.set_attribute("rag.embedding.model", self.embed_model_id)
            qvec = self.embed(question)

        with tracer.start_as_current_span("rag.retrieve") as span:
            hits = self.store.search(qvec, k=top_k)
            span.set_attribute("rag.retriever.top_k", top_k)
            span.set_attribute("rag.retriever.chunk_ids", [h.chunk_id for h in hits])
            span.set_attribute("rag.retriever.scores", [h.score for h in hits])
            span.set_attribute("rag.retriever.doc_versions", [h.doc_version for h in hits])

        with tracer.start_as_current_span("rag.assemble") as span:
            context = "nn".join(h.text for h in hits)
            span.set_attribute("rag.context.chunk_ids", [h.chunk_id for h in hits])
            span.set_attribute("rag.context.char_len", len(context))

        with tracer.start_as_current_span("rag.generate") as span:
            span.set_attribute("rag.llm.model", self.llm_model_id)
            span.set_attribute("rag.llm.prompt_version", self.prompt_version)
            out = self.llm.generate(question=question, context=context)
            span.set_attribute("rag.llm.prompt_tokens", out.prompt_tokens)
            span.set_attribute("rag.llm.completion_tokens", out.completion_tokens)
            return out.text

On a typical request the generate span dominates wall-clock time; retrieval is usually cheaper. Do not treat any published millisecond example from a vendor blog as your budget — measure your own p50/p95 per span, and put latency failure analysis on why your RAG pipeline is slow. Production collector wiring belongs on RAG observability.

How do you enable LangSmith tracing for a RAG app?

You enable LangSmith tracing by setting an API key and a tracing flag in the environment, then running the app — LangChain and LangGraph emit traces automatically; plain Python uses the LangSmith SDK. Versions and env names below are as of July 2026 (LangSmith docs; PyPI langsmith==0.10.10). Older tutorials still show LANGCHAIN_TRACING_V2=true; prefer the LangSmith-prefixed names going forward.

pip install langsmith==0.10.10

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="lsv2_..."
export LANGSMITH_PROJECT="rag-pipeline"   # optional; groups traces in the UI

For a non-framework function, wrap the entrypoint so each call becomes a root trace and nest retriever work as its own run:

from langsmith import traceable

@traceable(name="rag.query")
def rag_answer(question: str) -> dict:
    docs = retrieve(question)          # also @traceable(name="retrieve") if you want a child
    answer = generate(question, docs)
    return {"answer": answer, "chunk_ids": [d.id for d in docs]}

LangSmith is one managed backend. OpenTelemetry is the open export path into a collector you already run. Pick by where your traces already live — not by which blog ranked today. Where LangSmith sits among eval and observability products is scored on RAG evaluation tools.

What attributes must every RAG span carry?

Every RAG span must carry the attributes that make the four signatures decidable — without chunk_id, score and doc version on retrieve, and model plus prompt version on generate, the trace cannot diagnose the failure. The schema below follows practitioner guidance from Respan (May 2026) and Future AGI (2026); it is a field list, not a benchmark.

Do not store full chunk text in span attributes without PII redaction — Future AGI (2026) lists unredacted chunk payloads as a common mistake. Prefer ids plus a lookup into your document store. Drift across embedding or corpus versions is watched under why RAG gets worse over time.

How do you read a trace to find which stage failed?

You find the failing stage by opening the request’s root span and walking children in order until the first span whose output cannot support a correct answer. The walk:

1. Retrieve — are the expected chunk ids present, and are the top scores on-topic? If not, stop: you have irrelevant or incomplete retrieval.
2. Assemble — did those ids enter the prompt, or did truncation drop them? If they vanished here, the fault is context assembly, not the retriever.
3. Generate — does every claim in the answer appear in the assembled context? If the context is right and the answer invents or ignores it, you have context-ignored or hallucination.
4. Map the signature — match the row in the four-traces table and open the linked failure page for the fix.

That walk is the tracing half of debugging. The full ordered measurement procedure — what to assert per stage when you do not yet have a trace — is how to find which stage of your RAG pipeline broke.

What are common tracing pitfalls in RAG?

Common tracing pitfalls are the instrumentation choices that leave the four signatures undecidable. Six show up repeatedly across the live teardown:

• Generation-only traces — Braintrust (2026) notes that traces covering only the LLM call hide the retrieval decisions that usually explain bad answers. Always span retrieve separately.
• Span-per-chunk — one child per retrieved passage explodes storage and adds no diagnostic value; put chunk ids on the retrieve span (Future AGI, 2026).
• Missing doc or index version — without it, a stale corpus looks identical to a healthy one in the UI.
• Unpinned embedding, prompt or judge versions — a silent upgrade invalidates trendlines and creates false “drift”.
• PII in chunk attributes — treat retrieve spans like LLM message payloads; redact before export.
• Assuming OTel GenAI auto-instrumentation is enough — embedding, retrieval, rerank and prompt assembly are still dead zones unless you instrument them (Dev.to / erythix, 2026).

Which tools do RAG tracing and observability?

RAG tracing tools all export a span tree; they differ in UI, eval attachment and where the data lives. Three groups on the top-ranking results (as of July 2026) — this site sells none of them:

• Open-source tracers — Langfuse, Arize Phoenix — self-host or cloud; strong when you want the data under your control.
• Platform tracers — LangSmith, Braintrust, Galileo — managed UI plus online evaluators attached to traces.
• OTel into your APM — Elastic, Datadog, Dynatrace and similar collectors — useful when RAG traces must sit beside the rest of the service mesh.

A scored, tool-neutral comparison is the job of RAG evaluation tools compared. How to wire collectors, sampling and dashboards in production is RAG observability and tracing.