RAG

RAG Observability: Tracing Every Retrieval and Generation Step

Pramod Dutta · May 10, 2026 · 13 min read

Your RAG pipeline answered a question wrong yesterday. Was it the embedding model that missed the right chunk? Was it the reranker that buried a good result? Did the retriever return great context that the LLM simply ignored? Or was the chunk itself garbage because your PDF parser choked on a table? If you can't answer that in under five minutes, you don't have a RAG system — you have a black box with a chat interface bolted on. Most teams ship RAG with logging for the API layer and nothing for the reasoning layer, then wonder why every production incident turns into a two-hour archaeology dig through Slack screenshots. Observability is what turns that dig into a query.

Why RAG Breaks in Ways Normal Apps Don't

A typical web service fails loudly: 500 error, stack trace, done. RAG fails quietly. The pipeline returns a confident, well-formatted, completely wrong answer, and every component along the way reports success. The retriever found *something*. The reranker scored *something*. The LLM generated *something*. Nothing threw an exception, so nothing shows up in your error dashboard.

This is the core problem RAG observability exists to solve: correctness failures that look identical to success from the outside. You need visibility into the semantic content of each step, not just whether the step returned a 200.

A RAG pipeline typically has five or six places where things go wrong:

Ingestion: chunking splits a table mid-row, or a PDF parser drops a footnote that contained the actual answer.
Embedding: the query embedding lands in a different region of vector space than the relevant document because of a mismatched embedding model version.
Retrieval: the top-k results are topically related but don't contain the specific fact needed.
Reranking: a cross-encoder demotes the one chunk with the real answer because it's phrased differently than the query.
Context assembly: the final prompt truncates the retrieved context because someone forgot to account for token budget, silently dropping the last two chunks.
Generation: the LLM has the right context in front of it and still hallucinates, ignoring the retrieved passage entirely.

Six failure classes, and a single "the answer was wrong" bug report tells you nothing about which one fired. Tracing exists to collapse that six-way ambiguity into a two-minute lookup.

What "Tracing Every Step" Actually Means

Tracing isn't just wrapping your retrieval call in a try/except and logging the exception. A proper RAG trace is a structured, hierarchical record of the entire request lifecycle, where every stage records its inputs, outputs, and metadata as a span, and every span is a child of the overall request trace.

At minimum, a RAG trace should capture:

The raw user query, and if you rewrite or expand it (HyDE, multi-query expansion, query decomposition), every intermediate version.
The embedding call: model name, model version, embedding dimension, and latency.
The retrieval call: vector store queried, filters applied, top-k requested, and the actual documents returned with their similarity scores.
The reranking step, if you have one: the input order, output order, and the reranker's scores per document.
The assembled prompt: the literal string sent to the LLM, after all templating and truncation — not the template, the rendered output.
The generation call: model, temperature, token counts (prompt and completion), latency, and the raw response.
Any post-processing: citation extraction, guardrail checks, JSON parsing.

The key design decision is that every one of these is a span with a parent-child relationship to the overall trace, so you can look at one request end-to-end and see the full causal chain, not six disconnected log lines you have to correlate by timestamp.

Here's what that looks like using OpenTelemetry conventions, which is the substrate most RAG observability tools (Langfuse, Arize Phoenix, Traceloop, LangSmith) build on top of:

from opentelemetry import trace

tracer = trace.get_tracer("rag-pipeline")

def answer_question(query: str, session_id: str):
    with tracer.start_as_current_span("rag.request") as request_span:
        request_span.set_attribute("rag.query", query)
        request_span.set_attribute("session.id", session_id)

        with tracer.start_as_current_span("rag.embed_query") as embed_span:
            query_vector = embed(query, model="text-embedding-3-large")
            embed_span.set_attribute("embedding.model", "text-embedding-3-large")
            embed_span.set_attribute("embedding.dim", len(query_vector))

        with tracer.start_as_current_span("rag.retrieve") as retrieve_span:
            candidates = vector_store.search(query_vector, top_k=20)
            retrieve_span.set_attribute("retrieval.top_k", 20)
            retrieve_span.set_attribute(
                "retrieval.doc_ids",
                [c.id for c in candidates],
            )
            retrieve_span.set_attribute(
                "retrieval.scores",
                [round(c.score, 4) for c in candidates],
            )

        with tracer.start_as_current_span("rag.rerank") as rerank_span:
            reranked = rerank(query, candidates, top_n=5)
            rerank_span.set_attribute(
                "rerank.doc_ids", [r.id for r in reranked]
            )
            rerank_span.set_attribute(
                "rerank.scores", [round(r.score, 4) for r in reranked]
            )

        with tracer.start_as_current_span("rag.generate") as gen_span:
            prompt = build_prompt(query, reranked)
            gen_span.set_attribute("generation.prompt", prompt)
            response = llm.generate(prompt, temperature=0.1)
            gen_span.set_attribute("generation.completion", response.text)
            gen_span.set_attribute("generation.prompt_tokens", response.usage.prompt_tokens)
            gen_span.set_attribute("generation.completion_tokens", response.usage.completion_tokens)

        request_span.set_attribute("rag.final_answer", response.text)
        return response.text

Notice the pattern: every stage logs its own decision surface. When something goes wrong, you don't guess — you open the trace, look at retrieval.doc_ids and retrieval.scores, and immediately see whether the retriever even fetched the right document. If it did, you check rerank.scores to see if reranking demoted it. If it survived reranking, you read generation.prompt to confirm the chunk actually made it into the context window. If it's there, the bug is in the LLM's reasoning, not your retrieval stack. Each span eliminates one hypothesis.

Tracing Retrieval: Capture the Evidence, Not Just the Outcome

The retrieval span is the one teams under-instrument most often, because it's tempting to log only "retrieved 5 documents" and move on. That tells you nothing when a user reports a wrong answer.

What you actually need per retrieved chunk:

Document ID and source (which file, which page, which section).
Similarity score as returned by the vector store — cosine, dot product, or L2, whichever metric you're using.
Chunk text itself, or at least a hash and pointer to it, so you can inspect exactly what was retrieved without re-running the query days later against a vector store that may have since changed.
Filters applied — metadata filters (tenant ID, date range, document type) are a huge source of silent failures. A filter that's too aggressive returns zero results and nobody notices because the pipeline still "succeeds" with an empty context.
Retrieval latency, broken out separately from embedding latency, because a slow vector index is a different fix than a slow embedding API.

A subtle but critical detail: log the scores of documents that were retrieved *and* rejected, not just the final top-k. If your top-20 candidates before reranking never contained the right chunk, no amount of downstream tuning will fix the answer — you have a retrieval recall problem, and only the raw candidate list tells you that.

retrieve_span.set_attribute("retrieval.filters", str(metadata_filters))
retrieve_span.set_attribute("retrieval.candidate_count", len(candidates))
if len(candidates) == 0:
    retrieve_span.add_event("retrieval.empty_result", {
        "query": query,
        "filters": str(metadata_filters),
    })

That add_event call matters — an empty result set is exactly the kind of thing that should be flagged as an anomaly in your tracing UI, not buried as a normal-looking span with zero children.

Tracing Generation: Prompt, Completion, and the Gap Between Them

The generation span is where most of the actual "why is this wrong" investigation happens, and it's also the span people most often forget to fully instrument — usually because logging the entire rendered prompt feels wasteful or verbose. Don't skip it. The rendered prompt is the single most useful artifact in the entire trace, because it's the only place you can directly verify that the retrieved context actually reached the model in the form you intended.

Things that go wrong at this stage that only a full prompt log will reveal:

Silent truncation. You retrieved five great chunks, but your token-budget logic dropped the last two because someone hardcoded a context window size that's now stale after a model upgrade.
Ordering effects. Some models weight information at the start and end of the context window more heavily than the middle ("lost in the middle"). If your best chunk landed in position 3 of 5, that's worth knowing.
Template bugs. A Jinja template that renders {{ chunk.text }} instead of {{ chunk.content }} will silently insert empty strings for every chunk, and the LLM will confidently hallucinate an answer from its parametric memory instead of erroring out.
System prompt drift. If your system prompt is assembled dynamically (feature flags, A/B tests, per-tenant customization), you need the *exact* system prompt used for this specific request, not the current version in your repo.

Log completion metadata alongside the prompt:

gen_span.set_attribute("generation.model", "claude-sonnet-5")
gen_span.set_attribute("generation.temperature", 0.1)
gen_span.set_attribute("generation.stop_reason", response.stop_reason)
gen_span.set_attribute("generation.latency_ms", response.latency_ms)

stop_reason deserves special attention — if a completion stopped because it hit max_tokens instead of a natural stop, that's a strong signal your answer was cut off mid-thought, and no amount of prompt engineering will fix a token budget that's too small.

Correlating Traces Across a Multi-Turn Session

Single-request traces are necessary but not sufficient. Most real RAG products are conversational — a user asks a follow-up, and your system either re-retrieves with a rewritten query or reuses prior context. If your traces aren't linked by a session ID, you lose the ability to debug the most common class of chat RAG bug: context carried over incorrectly from a previous turn.

Tag every trace with a session_id and a turn_index, and make sure your tracing backend lets you view a session as a timeline, not just a flat list of independent requests.

request_span.set_attribute("session.id", session_id)
request_span.set_attribute("session.turn_index", turn_index)
request_span.set_attribute("session.parent_trace_id", previous_trace_id)

This matters enormously for query rewriting. If your pipeline does something like "combine the last two turns into a standalone query before embedding," you need to see the *rewritten* query in the trace, not just the user's literal message — because a bad rewrite ("what about the second one" turning into a garbled standalone query with no referent) is one of the single biggest causes of retrieval misses in conversational RAG, and it's invisible unless you log the intermediate step explicitly.

Building Dashboards That Surface Problems Before Users Report Them

Traces are for investigating a specific failure after you know about it. Dashboards are for finding out a failure class exists before a user has to tell you. The two are complementary, and skipping the dashboard layer means you're permanently in reactive mode.

Metrics worth aggregating across traces:

Retrieval score distribution. If your average top-1 similarity score drifts downward over a week, either your document corpus changed, your embedding model changed, or query patterns shifted — all things you want to catch before answer quality visibly degrades.
Empty-result rate. What percentage of queries return zero candidates after filtering? A spike here usually means a metadata filter bug or a document ingestion pipeline that silently stopped running.
Context utilization rate. What fraction of retrieved chunks actually appear, unmodified, in the final prompt? A dropping trend means your truncation logic is quietly discarding more context than it used to.
Token budget saturation. How often does the assembled prompt come within a few hundred tokens of the model's context limit? This tells you when truncation bugs are about to start happening, before they do.
Latency breakdown by span. Embedding, retrieval, reranking, and generation each have very different latency profiles and very different fixes. A dashboard that only shows "total request latency" hides which stage is actually the bottleneck.
Groundedness rate. What percentage of generated answers can be traced back to a specific retrieved chunk versus appearing to come from the model's parametric knowledge? This is usually computed with an automated judge, which is where evaluation and tracing start to overlap.

# Pseudocode for a nightly aggregation job over trace data
def compute_daily_rag_metrics(traces):
    empty_result_rate = sum(1 for t in traces if t.retrieval.candidate_count == 0) / len(traces)
    avg_top1_score = mean(t.retrieval.scores[0] for t in traces if t.retrieval.scores)
    truncation_rate = sum(1 for t in traces if t.generation.was_truncated) / len(traces)
    return {
        "empty_result_rate": empty_result_rate,
        "avg_top1_similarity": avg_top1_score,
        "truncation_rate": truncation_rate,
    }

Wire these into alerting the same way you'd alert on error rate or p99 latency. A RAG pipeline with a 15% empty-result rate is functionally broken for one in seven users, and without a dashboard, that fact sits invisible until support tickets pile up.

Instrumenting Without Drowning in Data

The obvious objection to all of this: logging full prompts, full completions, and every candidate document for every request generates an enormous volume of data, and most of it you'll never look at. Two practical patterns keep this manageable.

Sampling with escalation. Log lightweight summary spans (scores, latencies, token counts) for 100% of traffic, and log full-fidelity traces (complete prompts, complete completions, complete candidate lists) for a sampled percentage — say 5-10% — plus 100% of any request where something looks off: an error, an unusually low top-1 score, an unusually long latency, or a user-reported thumbs-down. This gets you cheap ambient visibility with expensive deep visibility exactly where you need it.

def should_capture_full_trace(request_span, sample_rate=0.05):
    if request_span.attributes.get("retrieval.candidate_count") == 0:
        return True
    if request_span.attributes.get("generation.was_truncated"):
        return True
    if request_span.attributes.get("user_feedback") == "negative":
        return True
    return random.random() < sample_rate

Redaction at the edge. If your documents contain PII or sensitive business data, don't solve this by skipping tracing — solve it by redacting before the trace is written, so the observability system never sees raw sensitive fields in the first place. Hash document IDs if the IDs themselves are sensitive, but keep the hash stable so you can still trace the same document across requests.

Tools That Do This Well

You don't need to build a tracing system from scratch — the OpenTelemetry-based ecosystem for LLM and RAG observability has matured a lot. A few worth knowing:

Langfuse — open source, purpose-built for LLM tracing, has first-class support for nested spans matching the retrieval-rerank-generate pattern, and includes built-in evaluation scoring on top of traces.
Arize Phoenix — strong on embedding-space visualization, which is genuinely useful for debugging retrieval quality (you can visually see whether a query embedding landed near the documents you expected).
LangSmith — tightly integrated if you're already using LangChain or LangGraph, with automatic span creation for chain and retriever calls.
Traceloop / OpenLLMetry — a thin OpenTelemetry instrumentation layer you can point at any backend (Datadog, Honeycomb, Grafana) if you already have observability infrastructure and don't want a separate LLM-specific tool.

Whichever you pick, the criteria that actually matter are: does it preserve the full hierarchical span structure, does it let you view a multi-turn session as a connected timeline, and does it make it easy to jump from an aggregate metric (like a dip in top-1 score) straight to the individual traces that caused it. A tool that just gives you a flat list of "LLM calls" without the retrieval and reranking spans attached is only solving half the problem.

From Tracing to Judging: Closing the Loop

Tracing tells you *what happened*. It doesn't tell you *whether it was good*. That's a separate, harder question, and it's where most teams stop too early — they build beautiful traces, stare at them manually for a few weeks, and never automate the judgment call of "was this answer actually correct and grounded."

This is where LLM-as-a-Judge comes in: once you have structured traces capturing the query, the retrieved context, and the generated answer, you can feed exactly those three fields into a separate judge model and ask it to score groundedness, relevance, and correctness on every single request, not just the ones a human happens to eyeball. The judge doesn't replace tracing — it consumes it. Your retrieval.doc_ids and generation.completion fields become the judge's input, and the judge's verdict becomes a new span you attach right back onto the same trace, so a low groundedness score takes you straight to the exact prompt and context that produced it. Get the tracing right first; the judge is only as good as the data you hand it.