LangSmith for RAG Debugging: Finding Retrieval Failures Fast

Your RAG app answers a question wrong, and your first instinct is to blame the model. You swap in a bigger LLM, rewrite the system prompt, crank the temperature down to zero, and the answer is still wrong. Here is the uncomfortable truth most teams learn only after weeks of prompt archaeology: the model never saw the right information in the first place. The retriever quietly handed it four irrelevant chunks, the LLM did its best with garbage, and everything downstream of that moment was doomed. LangSmith exists to make that invisible failure visible. It records every step of your pipeline as a trace, so instead of guessing which stage broke, you open the run, look at exactly which documents were retrieved, and know within seconds whether you have a retrieval problem or a generation problem. This guide walks through langsmith rag debugging end to end, from instrumenting your pipeline to building regression datasets from real failures.

Why RAG Bugs Are Almost Always Retrieval Bugs

A RAG pipeline is a chain of dependent stages: the user query gets embedded, the vector store returns the nearest chunks, an optional reranker reorders them, the chunks get stuffed into a prompt, and the LLM generates an answer grounded in that context. Every stage can fail, but the failures are not equally likely and not equally visible.

Generation failures are loud. When the model ignores the context or formats the answer badly, you can see it directly in the output, and prompt changes usually fix it. Retrieval failures are silent. The pipeline completes without errors, latency looks normal, the answer reads fluently, and the only symptom is that the content is wrong. The model was asked to answer a question about your refund policy using three chunks about your shipping policy, and it gamely produced a confident, wrong answer. Users call this a hallucination. It is really a retrieval miss wearing a hallucination costume.

This is why debugging RAG from the final answer alone is so slow. The final answer conflates two very different failure modes:

• The context contained the right information and the model failed to use it. This is a generation problem, fixable with prompting, model choice, or output constraints.
• The context never contained the right information. This is a retrieval problem, and no amount of prompt engineering will fix it, because you cannot prompt a model into knowing things it was never shown.

Until you can see the retrieved documents for a specific failing query, you cannot tell which of these happened. LangSmith gives you exactly that view: every retriever call in a trace shows the query that was embedded and the full text of every document that came back, with scores. The single most valuable habit in RAG debugging is this: when an answer is wrong, do not read the prompt first. Read the retrieved chunks first. In a large fraction of cases the investigation ends right there.

Instrumenting Your RAG Pipeline with LangSmith Tracing

LangSmith tracing is close to free to set up if you use LangChain, and only slightly more work if you built your pipeline by hand. With LangChain or LangGraph, tracing is enabled entirely through environment variables. No code changes are required:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="your-api-key"
export LANGSMITH_PROJECT="rag-support-bot"

Once these are set, every chain invocation is logged as a run tree in the named project. Retriever calls, LLM calls, prompt templates, and output parsers all show up as nested spans with their inputs and outputs captured.

If your pipeline is plain Python — you call an embedding API, query a vector database client, and hit an LLM SDK directly — you instrument it with the @traceable decorator from the LangSmith SDK. The important design decision is to give the retrieval step its own span with run_type="retriever", so LangSmith renders the returned documents in its dedicated document viewer instead of as a raw JSON blob:

from langsmith import traceable

@traceable(run_type="retriever", name="pinecone-retriever")
def retrieve(query: str, k: int = 4) -> list[dict]:
    vector = embed(query)
    hits = index.query(vector=vector, top_k=k, include_metadata=True)
    return [
        {
            "page_content": h["metadata"]["text"],
            "metadata": {"source": h["metadata"]["source"], "score": h["score"]},
        }
        for h in hits["matches"]
    ]

@traceable(name="rag-answer")
def answer(question: str) -> str:
    docs = retrieve(question)
    context = "\n\n".join(d["page_content"] for d in docs)
    return llm_call(question=question, context=context)

Because answer calls retrieve inside its own traced scope, LangSmith nests them automatically: one parent run for the whole request, one child run for retrieval, one child run for generation. That nesting is the whole point. When a user reports a bad answer, you find the parent run, expand the retriever child, and see precisely what the model was given.

Two practical tips for instrumenting well. First, include the retrieval score in each document's metadata, as shown above. Scores let you distinguish "the retriever confidently returned the wrong thing" from "the retriever returned weak matches because nothing good existed." Those are different bugs. Second, attach metadata to the parent run — user tier, document collection, prompt version, retriever configuration — because every metadata field becomes a filter you can slice by later.

Reading a RAG Trace: Where to Look First

Open any trace in LangSmith and you see a waterfall of spans down the left side and the selected span's inputs and outputs on the right. For RAG debugging there is a specific reading order that keeps you fast.

Start with the retriever span. Look at the query that was actually embedded, not the query you think was embedded. If your pipeline rewrites, condenses, or contextualizes the question before retrieval — most conversational RAG does — this is where you catch a rewriter that turned "what about for annual plans?" into a standalone question that dropped the actual topic. Query rewriting bugs are among the most common issues in multi-turn RAG, and they are invisible unless you look at the rewritten string.

Next, read the returned documents top to bottom, asking one question: does the information needed to answer correctly appear anywhere in these chunks? Be strict. A chunk that mentions the right product but not the right attribute does not count. If the answer is not present, you have confirmed a retrieval failure and you can stop reading the trace; the generation step is irrelevant.

If the answer is present in the context, move to the LLM span. Read the fully rendered prompt — not your template, the rendered string — and check where the relevant chunk landed. If it is buried at position four of six behind three distractor chunks, you may be looking at a ranking problem rather than a pure generation problem: the model anchored on the irrelevant material at the top. Then read the completion and see whether the model contradicted, ignored, or misread the context.

Finally, glance at the timing and token columns. A retriever span that returns in a few milliseconds with suspiciously round scores sometimes means you are hitting a cache with a stale index. A prompt whose token count has quietly doubled since last week usually means chunk sizes changed upstream. Traces carry operational signals as well as content.

This reading order — rewritten query, retrieved documents, rendered prompt, completion — turns a vague "the bot is wrong sometimes" report into a specific diagnosis in under two minutes per trace.

The Retrieval Failure Patterns You Will Actually See

After you have read a few dozen traces, retrieval failures stop looking random and start clustering into recognizable patterns. Knowing them ahead of time makes langsmith rag debugging dramatically faster, because you can pattern-match instead of reasoning from scratch.

1. The answer is not in the index. The retriever did its job; the document was never ingested, or was ingested before the content changed. In the trace, the top hits are topically adjacent but the specific fact is absent, and scores are mediocre across the board. The fix is an ingestion fix, not a retrieval fix.
2. The answer is in the index but chunked apart. The question needs a table row plus the table header, or a policy statement plus its exception clause, and your splitter put them in different chunks. In the trace you see a chunk that almost answers the question, cut off mid-thought. Fixes include larger chunks, overlap, or structure-aware splitting that respects headings and tables.
3. Vocabulary mismatch. The user says "cancel my plan," the docs say "terminate your subscription," and pure dense retrieval lands in the wrong neighborhood. Traces show low scores and semantically loose matches. Hybrid search that blends keyword and vector scores, or a query expansion step, usually resolves this.
4. Distractor dominance. The right chunk was retrieved at position five, but four near-duplicate chunks about a related topic outranked it, and with a top-k of four it got cut. You only catch this by temporarily raising k in a debug run and watching where the good chunk ranks. A reranker or deduplication at ingestion time is the fix.
5. Metadata filter mistakes. The query ran with a filter — tenant ID, language, document type — that silently excluded the right document. In the trace the results look inexplicably weak until you inspect the filter parameters on the retriever span. This one bites multi-tenant systems constantly.
6. Stale or duplicated index entries. The retriever returns an old version of a document alongside the new one, and the model averages the two into a confidently wrong answer. Traces show two chunks from the same source with conflicting content. The fix is upsert discipline and deleting superseded vectors.

Each pattern leaves a distinct fingerprint in the trace, and each has a fix that lives in a different part of your stack. This is the payoff of tracing: instead of one undifferentiated bucket called "bad answers," you get six actionable buckets, each with an owner.

Turning Bad Traces into a Regression Dataset

Finding a failure is only half the job. The other half is making sure the failure stays fixed after you change your chunking strategy, swap embedding models, or tune your reranker. In LangSmith, the mechanism for that is datasets built directly from traces.

Whenever you diagnose a genuinely bad run, add it to a dataset instead of just fixing it and moving on. In the UI you can select a run and add it to a dataset in two clicks, but doing it programmatically lets you fold it into your triage workflow:

from langsmith import Client

client = Client()

dataset = client.create_dataset(
    dataset_name="rag-retrieval-regressions",
    description="Real user queries where retrieval failed in production",
)

client.create_example(
    dataset_id=dataset.id,
    inputs={"question": "Can I get a refund after 30 days on an annual plan?"},
    outputs={
        "reference_answer": "Yes, annual plans have a 60-day refund window.",
        "expected_source": "policies/refunds-annual.md",
    },
)

Note what goes in the outputs: not just a reference answer but the expected source document. That second field is what lets you evaluate retrieval separately from generation. A generation-level eval asks "was the final answer right?" A retrieval-level eval asks "did the expected document appear in the retrieved set?" — and the second question is cheaper to score, faster to run, and points directly at the failing component.

Over a few weeks of triage this dataset becomes the most valuable artifact your team owns: a collection of the exact queries your real users asked that your system got wrong, with ground truth attached. Twenty to fifty hard examples curated from production traces will catch more regressions than a thousand synthetic questions generated from your docs, because synthetic questions tend to be phrased the way the documents are phrased — which is precisely the case retrieval is already good at. Your users' vocabulary mismatches, typos, and compound questions are the hard part, and only real traces capture them.

Scoring Retrieval Quality with Custom Evaluators

With a dataset in place, you can run experiments: execute your pipeline against every example and score the results automatically. LangSmith's evaluate function handles the orchestration; you supply the target function and the evaluators. The key move for retrieval debugging is to make your target function return the retrieved documents alongside the answer, so evaluators can score the retrieval step directly:

from langsmith import evaluate

def rag_target(inputs: dict) -> dict:
    docs = retrieve(inputs["question"])
    answer_text = answer(inputs["question"])
    return {
        "answer": answer_text,
        "retrieved_sources": [d["metadata"]["source"] for d in docs],
        "retrieved_texts": [d["page_content"] for d in docs],
    }

def source_recall(outputs: dict, reference_outputs: dict) -> dict:
    expected = reference_outputs["expected_source"]
    hit = expected in outputs["retrieved_sources"]
    return {"key": "source_recall", "score": 1.0 if hit else 0.0}

def context_precision(outputs: dict, reference_outputs: dict) -> dict:
    judged_relevant = sum(
        1 for text in outputs["retrieved_texts"]
        if llm_judge_relevance(text, reference_outputs["reference_answer"])
    )
    total = max(len(outputs["retrieved_texts"]), 1)
    return {"key": "context_precision", "score": judged_relevant / total}

evaluate(
    rag_target,
    data="rag-retrieval-regressions",
    evaluators=[source_recall, context_precision],
    experiment_prefix="hybrid-search-v2",
)

The two evaluators here measure complementary things. Source recall is a hard, deterministic check: did the document we know contains the answer show up at all? If recall is low, nothing downstream matters, and your work is in the retriever, the index, or the query rewriter. Context precision asks how much of what you retrieved was actually useful, using an LLM judge to grade each chunk against the reference. Low precision with high recall means the right material is arriving but drowning in noise — a reranking and top-k problem, and also a token-cost problem, since you are paying to stuff irrelevant chunks into every prompt.

Run this once to get a baseline, then rerun it with an experiment prefix every time you change anything in the retrieval path: chunk size, overlap, embedding model, hybrid weighting, reranker, k. LangSmith's comparison view puts experiments side by side, per example, so you can see not just that recall moved from one aggregate number to another but exactly which regression queries flipped from failing to passing — and, just as importantly, which previously passing queries broke. Retrieval changes are notoriously non-monotonic; a chunking change that fixes table questions often breaks narrative questions. Per-example comparison is how you catch the trade before your users do.

Add a faithfulness evaluator on the generation side too — an LLM judge that checks whether every claim in the answer is supported by the retrieved context. When faithfulness is high but answers are wrong, retrieval is guilty. When faithfulness is low, the model is embellishing beyond its context, and that is a prompting problem. The pair of scores keeps the blame pointed at the right component.

Debugging Retrieval in Production

Offline evaluation catches what you know to test. Production monitoring catches what you did not think of. LangSmith's project view over live traces supports both, and a few habits make it far more effective for RAG specifically.

First, wire user feedback into your traces. When your UI has a thumbs-down button, log it against the run ID with client.create_feedback(run_id, key="user_rating", score=0). Now the filter "user_rating is 0" in your project view is a standing queue of confirmed failures, each one a full trace with the retrieved documents attached. Triage becomes: open the trace, read the retriever span, classify against the failure patterns above, and add the worst offenders to your regression dataset. Fifteen minutes of this per day compounds into an unusually clear picture of how your system actually fails.

Second, use metadata filters to slice failures by configuration. If you tagged runs with retriever_version and chunking_strategy, you can compare feedback rates across configurations directly in the project view and catch a bad rollout early. This pairs naturally with gradual rollouts: run the new retriever for a fraction of traffic, tag it, and compare.

Third, set up automation rules for the failure signatures you cannot enumerate in advance. A rule that samples runs where the retriever's top score falls below a threshold catches vocabulary-mismatch queries as they happen. A rule that flags runs where the answer contains phrases like "I don't have information about" surfaces recall gaps — real user questions your index cannot answer yet, which is a content roadmap hiding inside your traces.

Finally, watch latency and cost per span, not just per request. Retrieval problems are sometimes performance problems: a reranker adding hundreds of milliseconds, or an over-eager k of twenty bloating every prompt. The same traces you use for correctness debugging carry the token counts and timings to catch these, and trimming retrieved context that your precision evaluator says is noise is one of the rare changes that improves quality and cost at the same time.

A Repeatable Workflow for Finding Retrieval Failures Fast

Everything above compresses into a loop you can run weekly, or daily during active development.

1. Collect. Tracing is on in every environment, retrieval has its own span with sources and scores in metadata, and user feedback is attached to runs. Without this, every other step is guesswork.
2. Triage. Work the negative-feedback queue and the automation-flagged runs. For each bad trace, read the rewritten query, then the retrieved documents, then the rendered prompt. Classify the failure: not indexed, chunked apart, vocabulary mismatch, distractor dominance, filter mistake, stale index, or genuine generation failure.
3. Capture. Every diagnosed failure becomes a dataset example with a reference answer and expected source. The dataset is append-only; it is your institutional memory of how your system fails.
4. Fix. Make one change at a time in the component the diagnosis points to. Resist the reflex to fix retrieval problems with prompt changes; it treats the symptom and masks the disease.
5. Verify. Run the experiment against the regression dataset, compare per-example against baseline, and confirm the targeted queries flipped without collateral damage elsewhere.
6. Ship and watch. Deploy behind a metadata tag, compare live feedback rates against the previous configuration, and fold any new failures back into step two.

The teams that debug RAG fast are not the ones with the fanciest retrievers. They are the ones for whom a wrong answer is a two-minute diagnosis instead of a two-day argument, because every request leaves a trace and every fix leaves a test. That discipline is what LangSmith buys you.

If you want to go deeper — full trace instrumentation for LangGraph agents, building LLM-as-judge evaluators that agree with human reviewers, prompt versioning, and production dashboards — our LangSmith Tutorial course on teachyou.ai walks through all of it hands-on, building a traced, evaluated RAG system from an empty repo to production monitoring. You will debug your last mystery hallucination the slow way, and every one after that the fast way.