DeepEval in Production: Continuous Monitoring Beyond CI

The gap between "tests pass" and "users are happy"

Every team that adopts DeepEval starts the same way: write a handful of LLMTestCase objects, wire up AnswerRelevancyMetric or FaithfulnessMetric, run deepeval test run in CI, and celebrate the green checkmark. That checkmark means something real — your prompt changes didn't break the fixed set of scenarios you thought to encode. But it means almost nothing about what happens six hours later when a user asks a question phrased in a way none of your test cases anticipated, your retriever returns a stale chunk because a document got re-indexed, or a downstream API starts timing out and your agent silently hallucinates a fallback answer.

CI evaluation is a snapshot. Production is a video that never stops playing. The moment your LLM application ships, the actual distribution of inputs, retrieved context, tool outputs, and model behavior diverges from whatever you tested against — sometimes gradually, sometimes overnight when a model provider pushes a silent update. Teams that stop at CI are flying blind for the 99% of a model's lifetime that happens after deployment. This article walks through how to extend DeepEval from a pre-merge gate into a continuous monitoring system: tracing production traffic, running online evaluations at scale, catching degradation before your users file support tickets, and closing the loop back into your test suite.

Why CI evaluation alone isn't enough

It's worth being precise about what CI evals can and can't catch, because the failure mode is subtle. A CI suite built with DeepEval typically looks like this:

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric

test_case = LLMTestCase(
    input="What's the refund window for annual plans?",
    actual_output="Annual plans can be refunded within 30 days of purchase.",
    retrieval_context=["Refund policy: annual subscriptions are refundable within 30 days."]
)

evaluate(
    test_cases=[test_case],
    metrics=[FaithfulnessMetric(threshold=0.7), AnswerRelevancyMetric(threshold=0.7)]
)

This is genuinely useful — it stops you from shipping a prompt change that breaks faithfulness on known cases. But three categories of failure live entirely outside this loop:

• Distribution drift. Your test set has 40 curated questions. Production sees thousands of real user phrasings, follow-up questions, and edge cases you never wrote down. A metric that scores 0.9 on your fixture set can score 0.5 on the long tail of real traffic.
• Non-deterministic degradation. LLM APIs change behavior without a version bump. Retrieval indexes get re-embedded and shift relevance. Vector database latency spikes and your retriever silently returns fewer chunks than expected. None of this shows up until real traffic hits it.
• Multi-turn and agentic drift. A single-turn test case can't reveal that your agent loses context after four conversation turns, or that a tool-calling agent occasionally picks the wrong tool when two tools have overlapping descriptions. These failure modes only emerge from real, messy conversations.

The fix isn't "write more test cases forever." It's accepting that evaluation needs to happen continuously, on live traffic, with the same rigor you apply in CI — and DeepEval's tracing and online evaluation features exist specifically for this.

Instrumenting your app with @observe

The foundation of production monitoring in DeepEval is the @observe decorator. It wraps any function in your LLM pipeline — a retriever call, an LLM invocation, a tool, an agent loop — and turns it into a span. The outermost decorated call becomes a trace, and nested calls form a tree underneath it, mirroring exactly how your application actually executes.

from deepeval.tracing import observe, update_current_span, update_current_trace

@observe(type="retriever", embedder="text-embedding-3-small")
def retrieve_context(query: str) -> list[str]:
    chunks = vector_store.search(query, top_k=5)
    update_current_span(
        input=query,
        retrieval_context=[c.text for c in chunks]
    )
    return [c.text for c in chunks]

@observe(type="llm", model="gpt-4o")
def generate_answer(query: str, context: list[str]) -> str:
    response = call_llm(query, context)
    update_current_span(input=query, output=response)
    return response

@observe(metric_collection="Production Support Bot")
def answer_question(query: str) -> str:
    context = retrieve_context(query)
    answer = generate_answer(query, context)
    update_current_trace(input=query, output=answer, retrieval_context=context)
    return answer

A few things matter here. First, this is non-intrusive — you're not rewriting your application logic to bubble up intermediate values for evaluation. You annotate the functions you already have, and DeepEval builds the trace tree around your existing control flow. Second, the type parameter classifies each span ("llm", "retriever", "tool", "agent", or a default general-purpose span), and each type accepts kwargs relevant to it — an "llm" span can track cost_per_input_token and cost_per_output_token, a "retriever" span can track top_k and chunk_size. This is what lets you later ask "which component in my pipeline is degrading" instead of just "is the final answer bad."

Third — and this is the part that makes tracing useful for monitoring rather than just debugging — update_current_span and update_current_trace populate the same fields as LLMTestCase (input, output, expected_output, retrieval_context, context, tools_called, expected_tools). Every span and every trace becomes a fully-formed test case, evaluable with the exact same metrics you already use in CI.

From tracing to online evaluation

Tracing alone gives you visibility — you can see what happened. Online evaluation is what turns that visibility into a signal you can act on. Once you're authenticated against Confident AI (deepeval login), the same @observe-decorated application streams every trace in real time, and you can attach a metric collection to run automatically against production traffic:

from deepeval.tracing import observe, update_current_trace

@observe(metric_collection="Production Support Bot")
def answer_question(query: str) -> str:
    context = retrieve_context(query)
    answer = generate_answer(query, context)
    update_current_trace(
        input=query,
        output=answer,
        retrieval_context=context,
        metric_collection="Production Support Bot"
    )
    return answer

The metric collection itself — which metrics run, at what thresholds — is configured once, and every subsequent invocation of your agent gets evaluated against it automatically. This is the crucial difference from CI: you are no longer deciding in advance which inputs to test. Every real user interaction becomes an evaluated data point, scored the moment it happens, without you writing a single new test case.

This matters because the metrics you already trust — FaithfulnessMetric, AnswerRelevancyMetric, HallucinationMetric, ContextualPrecisionMetric, and the full library of 50+ research-backed metrics — are the same ones running online. You're not maintaining two separate evaluation logics for "pre-release" and "post-release." A regression that would have failed your CI gate now gets flagged in production with the same definition of failure, just applied to real traffic instead of fixtures.

Component-level evaluation: finding where it broke, not just that it broke

One of the most common mistakes teams make with LLM monitoring is only evaluating the final output. If your end-to-end faithfulness score drops from 0.85 to 0.6, that tells you *something* is wrong — but not *what*. Was it the retriever pulling irrelevant chunks? A prompt regression in the generation step? A tool returning malformed data that got passed downstream?

Because @observe builds a full trace tree, you can attach metrics directly to individual spans, not just the trace root:

from deepeval.tracing import observe
from deepeval.metrics import ContextualRelevancyMetric, FaithfulnessMetric

@observe(
    type="retriever",
    embedder="text-embedding-3-small",
    metrics=[ContextualRelevancyMetric(threshold=0.7)]
)
def retrieve_context(query: str) -> list[str]:
    chunks = vector_store.search(query, top_k=5)
    return [c.text for c in chunks]

@observe(
    type="llm",
    model="gpt-4o",
    metrics=[FaithfulnessMetric(threshold=0.7)]
)
def generate_answer(query: str, context: list[str]) -> str:
    return call_llm(query, context)

Now when quality drops, your dashboard tells you which layer of the pipeline is responsible. A dip in ContextualRelevancyMetric on the retriever span while the LLM span stays healthy points squarely at your indexing or search logic — probably worth checking whether a recent document upload broke chunking, or whether an embedding model got swapped without updating the vector index. A dip in FaithfulnessMetric on the LLM span with healthy retrieval points at the generation step — maybe a prompt template got edited, or the model provider changed default behavior. This component-level granularity is what separates "we know something's wrong" from "we know what to fix and where."

Setting up alerts before your users complain

Continuous evaluation only pays off if someone — or something — is watching it. The overwhelming majority of production quality issues get discovered by users complaining, not by teams noticing dashboards. That's backwards, and it's fixable with threshold-based alerting.

Every metric in DeepEval carries a threshold (0.5 by default, but you should tune this per metric and per use case):

from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric

faithfulness = FaithfulnessMetric(threshold=0.75)
relevancy = AnswerRelevancyMetric(threshold=0.7)

When these metrics run online against production traffic, threshold breaches become the trigger for alerts. Configure alerting so that the moment aggregate faithfulness or relevancy dips below your bar — across a rolling window, not a single unlucky trace — your team gets notified. The practical goal is to shrink the gap between "quality regressed" and "someone competent knows about it" from days (when a support ticket eventually gets escalated) to minutes.

A pattern worth adopting: set two thresholds per metric, not one. A soft threshold that logs a warning and tags the trace for review, and a hard threshold that pages whoever owns the pipeline. Faithfulness dropping from 0.85 to 0.78 is worth a look during business hours. Faithfulness dropping to 0.4 across a chunk of traffic means something is actively broken — a bad deploy, an expired API key silently falling back to a worse model, a corrupted index — and it should wake someone up.

Full-trace evaluation with 100% coverage — not sampling

A natural instinct when volume gets large is to sample: evaluate 1 in 100 requests to control cost. Resist this instinct as your default. Sampling means you are explicitly choosing to be blind to most of your production traffic, and the failures that matter most — a bad actor probing for jailbreaks, a rare-but-catastrophic hallucination, a tool call that occasionally returns malformed JSON — are exactly the failures that sampling is most likely to miss, because they're rare by definition.

Running metrics against every trace, rather than a sample, changes the nature of what you can detect. It's the difference between "we think relevancy is roughly fine based on a sample" and "we can point to the exact three traces last Tuesday where the agent answered a billing question with pricing information from a different country's plan." When you're debugging a user-reported issue, having the specific trace — full input, retrieved context, intermediate spans, final output, and metric scores — is worth more than any amount of aggregated sampling data.

If cost genuinely is a constraint, the smarter lever isn't sampling traffic — it's tiering metrics. Run a cheap, fast metric (like a lightweight relevancy check) on every trace, and reserve expensive, deeply-reasoned metrics (like multi-step G-Eval chains) for traces that the cheap metric already flagged as suspicious. That preserves full coverage while controlling spend.

Multi-turn and conversational monitoring

Most production LLM applications aren't single-shot Q&A — they're conversations. A customer support agent, a coding assistant, an onboarding chatbot all maintain context across turns, and failure modes here are structurally different from single-turn failures. A model can answer turn one and turn two correctly, then lose track of a constraint the user set in turn one by turn five ("I'm vegetarian" gets forgotten three exchanges later).

DeepEval's tracing model handles this naturally because a trace can represent an entire conversation, with each turn as a nested span:

from deepeval.tracing import observe, update_current_trace

@observe(metric_collection="Conversational Agent Monitor")
def handle_conversation_turn(session_id: str, user_message: str, history: list) -> str:
    response = agent.step(user_message, history)
    update_current_trace(
        input=user_message,
        output=response,
        metadata={"session_id": session_id, "turn_count": len(history)}
    )
    return response

Tagging traces with session_id and turn_count in metadata lets you slice your monitoring dashboard by conversation length. It's common to see relevancy and faithfulness hold steady for the first two or three turns and then decay — that decay curve is invisible if you're only ever evaluating single-turn snapshots in CI, and it's exactly the kind of thing that turns into a churned customer before anyone on the team notices.

Closing the loop: turning production incidents into regression tests

The real payoff of production monitoring isn't the dashboard — it's what you do with what the dashboard finds. Every time an online evaluation flags a low-scoring trace, that trace is a gift: it's a real, naturally-occurring failure case that your CI test set didn't have. The discipline worth building into your team's workflow is converting flagged production traces directly into permanent regression tests.

from deepeval.test_case import LLMTestCase
from deepeval.dataset import EvaluationDataset

# Pull the flagged production trace and turn it into a permanent test case
regression_case = LLMTestCase(
    input="Can I get a refund if I'm on the annual plan but only used it for 2 months?",
    actual_output="No, annual plans are non-refundable after purchase.",
    retrieval_context=["Refund policy: annual subscriptions are refundable within 30 days."],
    expected_output="Yes, if within 30 days of purchase — otherwise no."
)

dataset = EvaluationDataset(test_cases=[regression_case])
dataset.add_test_case(regression_case)

Add this to your CI suite, and you've permanently closed the gap that let this failure through in the first place. Do this consistently — every incident becomes a test case — and your CI suite stops being a static snapshot written once at launch. It becomes a living document of every way your application has actually failed real users, which is a fundamentally stronger safety net than anything you could have anticipated up front. Teams that do this for six months end up with CI suites that catch the vast majority of regressions before merge, precisely because those suites are built from real production failures rather than a developer's imagination of what might go wrong.

Practical rollout: how to introduce this without boiling the ocean

If you're retrofitting monitoring onto an application that only has CI evals today, don't try to instrument everything on day one. A sensible rollout:

1. Start with the trace root only. Wrap your top-level "answer this request" function with @observe and get end-to-end traces flowing before you worry about component-level spans.
2. Pick two or three metrics that matter most for your use case. For RAG apps, that's usually faithfulness and contextual relevancy. For agents, tool correctness and task completion. Don't turn on all 50+ metrics at once — you'll drown in noise before you've calibrated thresholds.
3. Run in observe-only mode for a week before alerting. Let the metrics collect a baseline on real traffic. Your CI thresholds (tuned against a small fixture set) are often miscalibrated for the messier distribution of real traffic — you'll likely need to loosen some and tighten others.
4. Turn on alerting once thresholds are calibrated, starting with a single Slack or email notification channel rather than paging anyone.
5. Add component-level spans and metrics to your retriever, tool calls, and generation step once end-to-end monitoring is stable, so that when the aggregate score dips, you already have the instrumentation in place to find out why.
6. Build the feedback loop — a lightweight process (even a manual weekly review, at first) for pulling flagged low-scoring traces into your permanent test dataset.

This sequencing matters more than it looks. Teams that try to instrument every span, wire up every metric, and enable aggressive alerting in one sprint tend to abandon the whole effort a month later because it generated too much noise too fast. Monitoring that nobody trusts gets muted, and muted monitoring is worse than no monitoring, because it creates false confidence.

Wrapping up

CI evaluation answers "did this change break what I already knew to test." Production monitoring answers the much harder and more important question: "is this system working right now, for the actual people using it, in ways I never thought to write a test case for." DeepEval's tracing layer — the @observe decorator, span types, and update_current_span/update_current_trace calls — gives you the instrumentation to turn every real interaction into an evaluable test case, and online evaluations through Confident AI turn that instrumentation into continuous, full-coverage, alertable monitoring rather than a one-time report.

The teams that get the most value out of this aren't the ones with the most metrics enabled — they're the ones that treat every production incident as raw material for a stronger CI suite, closing the loop between what happens in the wild and what gets tested before the next deploy. That compounding effect, more than any single metric or dashboard, is what separates an LLM application that gets more reliable over time from one that just gets monitored more closely while quietly degrading.

If you want to go deeper on building this instrumentation from scratch — setting up tracing on a real RAG pipeline, configuring metric collections, tuning thresholds, and wiring up the incident-to-regression-test loop — our DeepEval Tutorial course on teachyou.ai walks through the entire production monitoring setup step by step, from your first @observe decorator to a fully alerting production dashboard.