Continuous Evaluation: Sampling Production Traffic for Ongoing QA

Your eval suite passed. Production is still on fire.

Here is a scene that plays out at almost every team shipping an LLM feature. The offline eval suite is green. Two hundred golden examples, hand-labeled, checked into CI, running on every pull request. The demo looks great. Then the feature ships, and three weeks later a support ticket says the assistant told a customer to dispute a charge that was never disputable. Nobody caught it because nobody was looking. The eval suite tested the cases the team thought of in January. The customer in March asked something nobody thought of.

This is the fundamental limit of static evaluation: it only knows about the inputs you gave it. Production doesn't work that way. Production sends you inputs in Portuguese when your eval set is all English. Production sends you a user who pastes a 40,000-token PDF into the chat box. Production sends you the edge case that turns your carefully tuned system prompt into a liability. A fixed test set, no matter how well-built, is a snapshot of what you anticipated. It cannot tell you about what you didn't.

Continuous evaluation is the practice of treating your live traffic as an ongoing, self-refreshing eval set. Instead of asking "did we pass our test suite," you ask "is a random slice of what real users are sending us still behaving well, this week, compared to last week." It's less glamorous than building a beautiful benchmark, and it's also the only method that actually detects drift, regressions, and unknown-unknowns after launch. This article is about how to build that pipeline: what to sample, how to log it safely, how to score it without needing a human to read every transcript, and how to turn the output into something your team actually acts on.

Why static test sets stop working the day you ship

Every eval set has an expiration date, even if nobody wrote it on the box. There are three separate forces that erode a fixed eval suite's usefulness over time, and it's worth naming them because each one calls for a different fix.

Distribution drift. The mix of things users ask changes. A support bot launched for billing questions slowly accumulates users asking about product features, then integrations, then "can you cancel my sister's account too." Your eval set, frozen at launch, has zero coverage of the traffic that now makes up 30% of volume.

Model and prompt drift. You update the system prompt to fix issue A. That change quietly degrades performance on scenario B, which was never in your eval set because scenario B was working fine before and nobody wrote a regression test for "don't break this." Underlying model providers also ship silent updates — a model alias like gpt-4o or claude-sonnet-4-5 can change behavior at the same pinned name across a provider-side update, and your last offline eval run was three sprints ago.

Adversarial and long-tail inputs. Real users are creative in ways your test-writers weren't able to anticipate. They'll paste code, switch languages mid-conversation, try prompt injection almost by accident, or ask something so oddly specific that no one on the team would have thought to write it as a fixture.

None of these show up in a static suite because a static suite, by definition, doesn't see new traffic. The only way to catch them is to keep evaluating on a rolling basis, using inputs your users actually generated, not the ones your team imagined.

What "continuous evaluation" actually means

Concretely, continuous evaluation is a loop with four stages that run on a schedule — hourly, daily, or per-deploy, depending on your traffic volume:

1. Sample a subset of live production requests and responses.
2. Log them, with enough context to reproduce and judge the interaction, while respecting privacy constraints.
3. Score the sample using automated evaluators — rule-based checks, embedding-based similarity, and LLM-as-a-judge grading.
4. Aggregate and alert, turning individual scores into trend lines, and firing a signal when quality drops below a threshold.

The point isn't to grade every single production interaction — that's expensive and often unnecessary. It's to have a defensible, statistically meaningful window into "how is this system actually behaving right now," refreshed continuously, instead of a single measurement taken at launch and never repeated.

It's useful to think of this as the production analog of a canary deployment. You don't manually watch every server in a fleet; you watch a sample of canary instances and infer the health of the rest. Continuous eval samples "canary conversations" out of your traffic and infers the health of the whole system from them.

Designing the sampling strategy

The naive approach is "log 1% of requests randomly." That's a reasonable default, but it under-serves the cases you actually care about most. A better sampling design layers several strategies together.

Uniform random sampling gives you an unbiased view of typical traffic. This is your baseline signal — if this line degrades, something broad is wrong.

Stratified sampling buckets traffic by dimension — user tier, language, feature flag, prompt template version, response length — and samples proportionally or with deliberate over-sampling of rare-but-important buckets. If 95% of your traffic is English and 5% is Spanish, uniform random sampling will barely ever catch a Spanish-language regression. Stratify so Spanish gets a guaranteed minimum sample size regardless of its share of volume.

Signal-triggered sampling captures traffic that looks risky even before you know if it's bad: unusually long responses, unusually long latency, a refusal keyword, a low confidence score from the model itself, or a user who sent a follow-up message like "that's wrong" or "no, I said." These signals are cheap to compute at request time and dramatically increase the density of interesting examples in your sample, compared to pure random sampling.

Outcome-triggered sampling captures traffic tied to a downstream signal: a thumbs-down click, a support ticket filed within the same session, a cart abandonment right after a product-recommendation response. This is the highest-value data you'll get, because it's pre-labeled by the real world.

A practical mix for a mid-volume product (say, 50,000–500,000 LLM calls a day) might look like: 2% uniform random, 100% of signal-triggered and outcome-triggered traffic (which is naturally rare, so it's cheap), and a stratification rule that guarantees at least 200 samples per day for every prompt template and every supported locale.

Here's what that sampling decision looks like as code, sitting in the response-handling path of a typical service:

import random
import hashlib

SAMPLE_RATE_DEFAULT = 0.02
RARE_LOCALES = {"pt-BR", "hi-IN", "sw-KE"}
MIN_DAILY_SAMPLES_PER_LOCALE = 200

def should_sample(request, response, daily_locale_counts):
    # Always capture anything with an explicit negative signal.
    if response.user_feedback == "thumbs_down":
        return True, "outcome_negative_feedback"

    if response.latency_ms > 8000:
        return True, "signal_high_latency"

    if response.finish_reason == "content_filter":
        return True, "signal_refusal"

    # Guarantee coverage for under-represented locales.
    locale = request.locale
    if locale in RARE_LOCALES:
        seen_today = daily_locale_counts.get(locale, 0)
        if seen_today < MIN_DAILY_SAMPLES_PER_LOCALE:
            return True, "stratified_locale_floor"

    # Deterministic hash-based sampling keeps the same
    # conversation consistently in or out of the sample
    # across turns, instead of re-rolling dice every message.
    digest = hashlib.sha256(request.conversation_id.encode()).hexdigest()
    bucket = int(digest[:8], 16) / 0xFFFFFFFF
    if bucket < SAMPLE_RATE_DEFAULT:
        return True, "uniform_random"

    return False, None

Note the hash-based bucketing on conversation_id rather than a fresh random.random() call per message. You want sampling decisions to be stable across a multi-turn conversation — if turn 1 is sampled, you generally want turns 2 and 3 sampled too, because a judge model evaluating a single out-of-context turn is far less reliable than one evaluating a full thread.

Logging without becoming a privacy incident

Once you decide to keep a request, you have to store it, and this is where teams get sloppy in ways that come back to bite them. A few non-negotiable rules:

Redact before you persist, not after. PII scrubbing that happens as a batch job "later" means there's a window where raw PII sits in a data store, and it means every downstream consumer of that store (your eval dashboard, your judge model calls, anyone with read access) is exposed until the batch job runs. Redact — or at minimum tokenize — at write time.

Separate the eval store from the primary application database. You don't want your QA sampling pipeline to be a load-bearing dependency of your product's critical path, and you don't want an eval engineer's read query locking a production table.

Log the full context needed to judge, not just the final message. A judge (human or model) evaluating "was this response correct" needs the system prompt version, the retrieved documents if you're doing RAG, the tool calls and tool outputs, and the conversation history — not just the last assistant turn in isolation. Teams that log only the final output frequently can't reproduce why a judge flagged something, because the missing context was the whole story.

Version everything. Prompt template version, model name and version, retrieval index version, feature flag state. Six weeks from now when a trend line dips, "which prompt version was live on that date" is the first question anyone asks, and if you didn't log it, you're reconstructing it from deploy timestamps and hoping they line up.

A minimal log record for a RAG-backed assistant might look like this:

{
  "conversation_id": "conv_8f3a...",
  "turn_id": "turn_4",
  "timestamp": "2026-07-03T14:22:01Z",
  "sample_reason": "outcome_negative_feedback",
  "prompt_template_version": "support-v14",
  "model": "claude-sonnet-4-5-20250929",
  "locale": "en-US",
  "retrieved_doc_ids": ["kb_2291", "kb_0447"],
  "tool_calls": [
    {"name": "lookup_order", "args": {"order_id": "REDACTED"}, "result_summary": "found, status=shipped"}
  ],
  "user_message_redacted": "my order [REDACTED_ORDER_ID] hasn't arrived",
  "assistant_response": "I can see your order shipped on...",
  "user_feedback": "thumbs_down",
  "latency_ms": 1840
}

Store PII-bearing fields as references (REDACTED, or a reversible token behind an access-controlled lookup service) rather than raw values, so the eval dataset itself can be shared more broadly across the team — including with whoever is writing judge prompts — without expanding the blast radius of a data leak.

Scoring the sample: rules, embeddings, and judges

Once you have a steady stream of sampled, redacted transcripts, you need to score them without a human reading every one. Three layers, roughly cheapest to most expensive:

Deterministic checks catch the mechanical failures: did the response contain a required disclaimer, is the JSON output actually valid JSON, does the response length fall within expected bounds, did a tool call that should have fired actually fire. These are cheap regex or schema checks and should run on 100% of sampled traffic, not just a subset.

import json

def deterministic_checks(record):
    results = {}

    if record.get("expects_json_output"):
        try:
            json.loads(record["assistant_response"])
            results["valid_json"] = True
        except json.JSONDecodeError:
            results["valid_json"] = False

    response_len = len(record["assistant_response"])
    results["length_in_bounds"] = 20 <= response_len <= 4000

    if record.get("requires_disclaimer"):
        results["has_disclaimer"] = "not financial advice" in record["assistant_response"].lower()

    return results

Embedding and similarity checks are useful for catching drift in tone or topic without needing a full LLM call — comparing the embedding of today's responses against a baseline cluster of "known good" responses from launch, and flagging outliers for closer review. This is a cheap first-pass filter to decide which transcripts are worth sending to the more expensive judge stage.

LLM-as-a-Judge is where the real qualitative grading happens — helpfulness, correctness against retrieved context, tone, safety, whether the response actually answered the question asked. This is more expensive per-item than the first two layers, which is exactly why sampling matters: you cannot afford to run a judge model over 100% of production traffic at scale, but you can absolutely afford to run it over a well-chosen 2-5% sample plus 100% of the outcome-triggered subset.

A common early objection to this whole approach is "we can't afford to judge that much traffic." In practice, the math is friendlier than it looks, because judging is decoupled from serving — you're not adding a judge call to the user-facing request path, you're running a batch job against logged samples afterward, and you control exactly how much of that batch job you're willing to pay for. Work backward from the question you're actually trying to answer. If you want to detect a drop in quality from, say, 90% pass rate to 80% pass rate for a given prompt template, with reasonable statistical confidence, a few hundred judged samples per day per template is usually enough to see the shift clearly in a rolling average — you don't need tens of thousands. For a product running ten distinct prompt templates, a daily judge budget of two or three thousand calls is often plenty, even at high traffic volume, because the sampling layer's whole job is to decouple judge cost from raw traffic volume.

Cost isn't uniform across the three scoring layers, and that's deliberate. Deterministic checks are essentially free and should run on the full sample. Embedding similarity checks are cheap per call and a good filter to shrink what reaches the judge. LLM-as-a-Judge calls are the expensive layer, so they should only ever see what survived the cheaper filters or what was already flagged by a strong outcome signal like a thumbs-down. Teams that skip straight to "judge everything with a top-tier model" burn budget re-discovering things a regex would have caught for a fraction of a cent, and that's usually the actual reason continuous eval gets shelved as "too expensive" — not because the technique doesn't scale, but because the layering was skipped.

It's also worth revisiting the sample rate periodically rather than setting it once. Early in a feature's life, when traffic is low and every conversation is precious signal, a much higher sample rate — even 100% — is affordable and valuable. As volume grows, the rate can shrink because the absolute number of samples at a lower percentage still stays statistically sufficient. Treat the sample rate as a dial you turn based on current traffic and current budget, not a constant you set at launch and forget.

Turning judge output into a trend line, not a pile of transcripts

A judge call that returns a single number for a single transcript is not yet useful. The useful unit is a trend: how has the average faithfulness score for the "order status" prompt template moved over the last 14 days, broken out by locale. Aggregation turns individual scores into a signal a team can act on without reading every transcript.

from collections import defaultdict
from datetime import datetime, timedelta

def rolling_quality_trend(scored_records, window_days=14):
    by_day_and_template = defaultdict(list)

    for r in scored_records:
        day = r["timestamp"][:10]
        key = (day, r["prompt_template_version"])
        by_day_and_template[key].append(r["judge_score"])

    trend = {}
    for (day, template), scores in by_day_and_template.items():
        trend.setdefault(template, {})[day] = {
            "mean": sum(scores) / len(scores),
            "n": len(scores),
            "p10": sorted(scores)[max(0, len(scores) // 10)],
        }
    return trend

Track the p10 (a lower percentile), not just the mean. A mean can hold steady while the bottom decile quietly gets worse — a growing tail of bad responses hidden behind a mass of fine ones. That tail is usually where the support tickets come from.

Set alert thresholds relative to a trailing baseline rather than an absolute number, since "good" is domain-specific and drifts naturally with product changes. A reasonable rule: alert if the 7-day rolling mean for a given prompt template drops more than one standard deviation below its own 30-day baseline, or if judged-unsafe rate for any template exceeds a hard ceiling regardless of trend (safety thresholds should be absolute, not relative — you don't want a slow-drifting baseline to normalize an increasing rate of harmful outputs).

Closing the loop: from alert to fix

A continuous eval pipeline that only produces dashboards nobody looks at is wasted engineering effort. The loop has to close somewhere concrete:

• Route flagged transcripts into a triage queue. Whoever owns the prompt template gets a weekly (or, for high-severity flags, same-day) queue of the worst-scoring sampled transcripts, with the judge's rationale attached, not just the score.
• Promote recurring failures into the offline eval set. This is the mechanism that actually closes the loop between "we saw this go wrong in production" and "we will never regress on this specific case again." Every distinct failure pattern found via continuous eval should become a new fixture in the CI-gated eval suite, so future prompt or model changes are tested against it before they ship, not after.
• Track judge disagreement with humans. Periodically have a human re-grade a subset of what the judge already scored, and measure agreement. If judge-human agreement drops, that's a sign the judge prompt itself has drifted out of sync with what "good" means for your product, and needs recalibration — not the underlying application.
• Version the judge prompt too. It is easy to forget that your judge is itself a prompt against a model, subject to the exact same drift problems as your application prompt. Log which judge prompt version produced which score, so a jump in scores can be correctly attributed to "the app got better" versus "we changed how we're grading it."

Start small, then let it earn its complexity

You don't need the full four-stage pipeline on day one. A reasonable rollout order: start by logging 100% of thumbs-down and error-flagged interactions with full context — that alone, reviewed weekly, catches a surprising share of real issues for almost no engineering cost. Add uniform random sampling at 1-2% once you have a redaction pipeline you trust. Add deterministic checks next, since they're nearly free to run at full sample volume. Only bring in LLM-as-a-Judge scoring once you have enough sampled volume that a human can no longer keep up with manual review — that's the actual signal that you've outgrown ad hoc QA and need the judge layer to keep pace with your traffic.

The teams that get burned by production LLM failures are rarely the ones without any evaluation at all — they're the ones who built a solid eval suite once, at launch, and then treated it as finished. Continuous evaluation is the admission that "finished" isn't a real state for this kind of system. Traffic changes, models change, and the only way to know your system is still behaving the way you think it is, is to keep checking it against what users are actually sending — automatically, on a schedule, with a LLM-as-a-Judge doing the qualitative grading your team no longer has time to do by hand.