Offline vs Online LLM Evals: What to Run Where

Offline evals score your LLM system against a fixed dataset before you ship. Online LLM evaluation scores live production traffic after you ship, using LLM judges, inline guardrails, and user feedback signals instead of ground-truth labels. You need both, because they answer different questions: offline evals tell you whether a change is safe to deploy, and online evaluation tells you whether the system actually works for real users on inputs you never thought to test. This guide maps each type of check to the right side of that line, with runnable code for both, and shows how to pipe production failures back into your offline suite so the whole eval program compounds instead of rotting.

What Separates Offline and Online LLM Evals

An offline eval runs against a curated dataset with known expectations. The inputs are fixed, so scores are comparable across runs: you can diff prompt v14 against prompt v15 on the exact same 200 cases and know that any score movement came from your change, not from traffic drift. Offline evals run on demand: in CI on every pull request that touches a prompt, nightly against the full suite, or ad hoc when you are comparing two models before a migration.

Online LLM evaluation scores real requests as they flow through production. There is no ground truth, because nobody has labeled the question a user typed thirty seconds ago. Instead you score proxies: does the answer stay grounded in the retrieved context, did the user click regenerate, did the conversation end in an escalation to a human. Online scores drift with traffic mix, so absolute numbers matter less than trends over time and deltas between experiment arms.

The closest software analogy: offline evals are your unit and integration tests, online evaluation is observability with a quality layer on top. But the balance is different from classical software. A REST API has a bounded input space defined by its schema, so tests catch most defects. An LLM application accepts anything a human can type. The input distribution is discovered in production, not designed in a spec, which is why teams that only run offline evals consistently get surprised, and teams that only run online evaluation ship regressions they could have caught for free.

Concretely, for a support bot: the offline suite is 200 curated tickets with expected resolutions, run in CI. The online layer samples a slice of live conversations every hour and has an LLM judge score groundedness and resolution quality, while a dashboard tracks regenerate clicks and escalation rate per prompt version.

What Offline Evals Are Good For

Offline evals are the only place you get controlled comparisons. Use them for:

• Regression gating. You changed the system prompt, swapped the model, or re-chunked your RAG corpus. Run the suite, compare against the baseline from main, block the merge if the pass rate drops. This is the single highest-value eval habit a team can build.
• Candidate comparison. Deciding between two models or two prompt strategies requires identical inputs for both candidates. Only a fixed dataset gives you that. Model migrations (say, moving a pipeline to claude-sonnet-5 or claude-opus-4-8) should never happen without an offline pass first.
• Structured output contracts. JSON schema validity, enum adherence, tool-call argument correctness. These are cheap, deterministic assertions with zero judge cost.
• Safety and red-team suites. Jailbreak attempts, prompt injection payloads, PII extraction probes. You cannot wait for these to show up organically in production, and you do not want your first data point to be an incident.
• Component metrics for RAG. Retrieval recall against labeled query-document pairs, chunking quality, reranker lift. These need labels, so they live offline.

What offline evals structurally miss: distribution shift, new user intents you did not anticipate, upstream content changes that silently break retrieval, latency and cost under real concurrency, and everything about user satisfaction. A golden dataset is a snapshot of what you knew when you wrote it. Every product change and every shift in your user base makes some slice of it stale, which is why the loop described later in this article matters more than the initial dataset.

What Online LLM Evaluation Catches That Offline Cannot

Online LLM evaluation is how you find out what your system actually does, as opposed to what it does on the cases you thought of. It adds four things no offline suite can provide.

First, real inputs. Production traffic is the eval set you did not know how to write: misspelled questions, mixed languages, users pasting entire log files, questions about a feature you shipped yesterday. Sampled judge scores over that traffic measure quality on the true distribution.

Second, continuous measurement. Offline runs are points in time. Online evaluation is a curve. When an upstream API changes its response format and your retrieval quietly degrades, a moving average of groundedness scores catches it in hours. Your offline suite would have caught it at the next release, days later, if the golden set happened to cover that path.

Third, implicit user feedback. Thumbs up and down are sparse and biased, but production gives you richer signals for free: regenerate clicks, copy events, whether the user rephrased the same question, whether a session ended in escalation or abandonment. These are weak labels individually and powerful in aggregate.

Fourth, the business tie-in. Deflection rate, task completion, handle time, conversion. Quality scores that never connect to a business metric eventually lose the argument for headcount and budget. Online evaluation is where that connection gets made.

The costs are real too. Judge inference costs money at traffic scale, which is why sampling matters. There is no ground truth, so judge bias and rubric quality directly cap how trustworthy your scores are. And online evaluation cannot block a bad deploy: by the time a score dips, users already saw the regression. It shortens detection time, it does not prevent the incident. That is the offline suite's job.

The Decision Matrix: What to Run Where

The practical question is never "offline or online", it is "which checks belong where". Use this mapping:

• Format and schema validation -> both. Offline as hard assertions in CI, online as inline guardrails on every request, since code-based checks cost nothing.
• Exact correctness against golden answers -> offline only. There is no golden answer for live traffic.
• Groundedness and hallucination -> both. Offline against cases with labeled context, online via sampled LLM-judge scoring with the retrieved context stored in the trace.
• Safety and policy compliance -> both, asymmetrically. A red-team suite runs offline before every release. A fast moderation classifier runs inline on 100 percent of production traffic, with an async judge auditing a sample for subtler violations.
• Retrieval quality -> offline for recall against labeled pairs, online via proxies like judge-scored context relevance.
• Latency, cost, and token usage -> online. Offline load tests only approximate real concurrency and real input lengths.
• User satisfaction and task completion -> online only. Nothing offline predicts this reliably.
• Prompt and model comparisons -> offline first for coarse ranking, then an online experiment on the survivors to make the final call.
• Multi-turn conversation quality -> mostly online, because simulating realistic multi-turn users offline is expensive and usually unconvincing. Keep a small simulated-user smoke suite offline for the flows that must never break.

The rule of thumb behind all of this: if a check is cheap and deterministic, run it everywhere. If it needs ground truth, it lives offline. If it needs real users, it can only live online.

Building an Offline Eval Suite That Gates Deploys

Keep the dataset in git next to the prompts it tests. JSONL works well because diffs are reviewable and appends never conflict:

{"id": "refund-30d", "input": "Can I get a refund after 45 days?", "must_contain_facts": ["refunds are only available within 30 days of purchase"], "must_not_contain": ["full refund"]}
{"id": "sso-plan-gate", "input": "How do I enable SAML SSO?", "must_contain_facts": ["SSO requires the Business plan"], "must_not_contain": []}

The harness is plain pytest, which means it runs in CI with zero new infrastructure:

import json
import pathlib

import pytest

from myapp.pipeline import answer          # your real entrypoint
from evals.judge import judge_contains_fact

CASES = [
    json.loads(line)
    for line in pathlib.Path("evals/golden.jsonl").read_text().splitlines()
    if line.strip()
]

@pytest.mark.parametrize("case", CASES, ids=lambda c: c["id"])
def test_golden_case(case):
    output = answer(case["input"])

    for fact in case["must_contain_facts"]:
        assert judge_contains_fact(output, fact), f"missing fact: {fact}"

    for banned in case["must_not_contain"]:
        assert banned.lower() not in output.lower(), f"forbidden text: {banned}"

Deterministic assertions (the must_not_contain check) need no judge. For semantic checks, a small LLM judge with a binary rubric is cheap and surprisingly reliable:

import anthropic

client = anthropic.Anthropic()

JUDGE_PROMPT = """You are grading a customer support answer.

Required fact: {fact}

Answer to grade:
{answer}

Does the answer clearly state the required fact? Reply with exactly one word:
PASS or FAIL."""

def judge_contains_fact(answer: str, fact: str) -> bool:
    response = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=4,
        messages=[{
            "role": "user",
            "content": JUDGE_PROMPT.format(fact=fact, answer=answer),
        }],
    )
    verdict = next(b.text for b in response.content if b.type == "text")
    return verdict.strip().upper().startswith("PASS")

Four practices separate suites that survive from suites that get deleted:

1. Gate on deltas, not absolutes. Fail CI when the pass rate drops more than a threshold versus main, rather than demanding 100 percent. LLM outputs have variance, and a suite that flakes gets ignored within a month.
2. Cache judge calls. Key the cache on the tuple of judge model, judge prompt, and inputs. Unchanged cases cost nothing on re-runs, which keeps PR feedback fast and the CI bill small.
3. Pin and calibrate the judge. Pin the judge model version so scores stay comparable across weeks. Then evaluate the judge itself: hand-label 50 outputs, measure agreement between judge and human, and fix the rubric until agreement is boring. An uncalibrated judge is a random number generator with a paycheck.
4. Split the suite. A fast smoke subset on every PR, the full suite nightly and before releases. Binary verdicts per case beat 1-to-10 scores for regression gating, because a diff of PASS to FAIL on case refund-30d is actionable and a drop from 7.8 to 7.4 is not.

If you would rather not hand-roll the harness, promptfoo and DeepEval cover this pattern well, Braintrust and LangSmith add hosted baselines and diff views, and Ragas ships reference-based RAG metrics. The architecture stays the same regardless of tool: dataset in git, assertions plus judges, delta gate in CI.

Setting Up Online LLM Evaluation in Production

Online LLM evaluation is three layers, and it fails if you skip the first one.

Layer 1: tracing. Every request gets a trace with the user input, retrieved context, full prompt, prompt version, model id, output, latency, token counts, and session id. Without the retrieved context in the trace, a groundedness judge has nothing to check against, and a bad score is undebuggable. OpenTelemetry has GenAI semantic conventions for exactly this, and Langfuse, Arize Phoenix, LangSmith, and W&B Weave all ingest traces in this shape. Langfuse and Phoenix are open source and self-hostable, which matters once traces contain customer data.

Layer 2: inline guardrails. Blocking checks that run on 100 percent of requests and finish in milliseconds: schema validation on structured outputs, a moderation classifier, regex-level PII detection, max-length limits. These are code, not judges. Never put an LLM judge in the blocking path of a user request; it adds seconds of latency and a new failure mode to every call.

Layer 3: async judges. A background worker samples traces, scores them against rubrics, and writes scores back to the trace store. Nothing user-facing waits on it.

import random

import anthropic

client = anthropic.AsyncAnthropic()

SAMPLE_RATE = 0.05   # score 5 percent of traffic

RUBRIC = """Score this support answer for groundedness.

Retrieved context:
{context}

User question:
{question}

Answer given to the user:
{answer}

A 5 means every claim is supported by the context. A 1 means the answer
contradicts the context or invents facts. Reply with a single digit, 1 to 5."""

async def maybe_score_trace(trace: dict) -> None:
    if random.random() > SAMPLE_RATE:
        return
    response = await client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=4,
        messages=[{
            "role": "user",
            "content": RUBRIC.format(
                context=trace["retrieved_context"],
                question=trace["user_input"],
                answer=trace["model_output"],
            ),
        }],
    )
    text = next(b.text for b in response.content if b.type == "text")
    await store_score(
        trace_id=trace["id"],
        metric="groundedness",
        value=int(text.strip()[0]),
        judge_model="claude-haiku-4-5",
        prompt_version=trace["prompt_version"],
    )

Feed this worker from a queue (Celery, SQS, a Postgres outbox, whatever you already run) so a judge outage never touches the request path. Store the judge model and prompt version alongside every score, because you will change both and need to know which scores are comparable.

Wire in the implicit feedback signals too: log regenerate clicks, thumbs, copy events, and human escalations as events attached to the trace. These are your cheapest labels, and traces with negative feedback deserve a judge score at 100 percent sampling rather than 5 percent.

Finally, alert on trends, not points. A seven-day moving average of groundedness per prompt version, with an alert on a sustained drop, catches real regressions without paging anyone over one bad answer. Add embedding-based drift detection on inputs if you want early warning that a new intent is arriving before quality scores move. And if traces leave your infrastructure for judging, strip PII first and check your data processing agreements; scoring user conversations with a third-party judge is a data flow your privacy review needs to see.

Running Online Experiments Without Burning Users

Once judges and feedback signals exist, A/B testing prompts and models becomes routine. Hash the user id to an experiment arm so a user always sees the same variant, and treat the prompt version as the treatment. Compare judge scores, negative-feedback rate, task completion, p95 latency, and cost per request between arms.

Two rules keep this safe. First, nothing enters an online experiment without passing the offline gate; online experiments rank survivors, they are not how you discover catastrophes. Second, for risky changes use shadow mode: run the candidate on a mirrored copy of traffic, score both outputs asynchronously, and let users see only the control. Shadow mode costs double inference on the mirrored slice and is worth every token for model migrations.

Closing the Loop: Production Failures Become Offline Tests

This is the part most teams skip, and it is the actual point of running both kinds of evals. Set up a weekly triage over the traces your online layer flagged:

select t.id, t.user_input, t.model_output, s.value as groundedness
from traces t
join scores s on s.trace_id = t.id
where s.metric = 'groundedness'
  and s.value <= 2
  and t.created_at > now() - interval '7 days'
order by t.created_at desc;

Pull the low-scored and negative-feedback traces, dedupe them (embedding clustering helps when volume is high), hand-label a manageable batch, and append the confirmed failures to your golden JSONL with correct expectations. Each production failure becomes a permanent regression test. Retire cases when the product intentionally changes behavior, and version the dataset so score history stays interpretable.

Run the loop for a few months and the character of the offline suite changes: it stops being a list of cases the prompt author imagined and becomes distilled production reality. That is when offline scores start actually predicting online outcomes, because the offline distribution finally resembles the real one. Recalibrate your judges against fresh human labels on the same cadence, since traffic drift quietly invalidates judge rubrics too.

Common Mistakes in Offline and Online Evaluation

• Scoring 100 percent of traffic with a frontier judge. The insight per dollar is terrible. Sample a small slice broadly, judge negative-feedback traces fully, and spend the savings on better rubrics and human calibration.
• Scores without traces. A groundedness of 2 with no stored context and prompt version is trivia. Every score must link to a full trace or it cannot be debugged.
• Judging the generator with itself and never checking. Self-preference bias is real. Use a different model family for the judge where practical, and always validate the judge against human labels regardless of which model judges.
• A fifteen-case offline suite written by the prompt author. It passes forever and certifies nothing. Seed the suite from real tickets and grow it from production failures.
• Treating the latency dashboard as quality monitoring. A system can be fast, cheap, up, and wrong.
• Blocking requests on judge calls. Inline checks must be milliseconds of code. Judges are async, always.
• Comparing online scores across quarters without checking traffic mix. Online numbers are only comparable when the distribution is, which is another reason experiment arms beat historical baselines.
• Never re-baselining after intentional changes. When you rewrite the refund policy, update the golden cases that encode the old one, or your suite starts failing for the wrong reasons and gets ignored.

A One-Week Rollout Plan

1. Day 1: Put prompts under version control if they are not already. Write 50 golden cases sourced from real tickets and known past failures.
2. Day 2: Stand up the pytest harness with deterministic assertions plus one binary judge. Wire the delta gate into CI.
3. Day 3: Add tracing. Self-host Langfuse or Phoenix, or start with a plain Postgres table; the schema above is enough.
4. Day 4: Deploy the async judge worker at a low sampling rate with one rubric, groundedness first if you run RAG. Log regenerate clicks and thumbs as trace events.
5. Day 5: Build the moving-average dashboard per prompt version, set one alert, and put the weekly triage meeting on the calendar. The triage is the flywheel; protect it.

From there, everything is iteration: more rubrics, better calibration, shadow mode for the next model migration, and a golden set that grows a little every week.

FAQ

What is online LLM evaluation?

Online LLM evaluation is the practice of measuring an LLM application's output quality on live production traffic, typically by sampling requests and scoring them with LLM judges against rubrics like groundedness or answer relevance, combined with inline guardrails and implicit user feedback signals such as regenerate clicks and escalations. It complements offline evals, which run on curated datasets before deployment.

Can online LLM evaluation replace offline evals?

No. Online evaluation cannot block a bad deploy, because by the time scores move, users have already seen the regression. It also lacks ground truth and controlled comparisons, so it cannot tell you cleanly whether prompt A beats prompt B on identical inputs. Offline evals gate changes; online evaluation verifies reality and finds the failures your dataset missed.

How many examples should an offline eval set have?

Start with 50 to 150 cases sourced from real user inputs and known failures, not from the prompt author's imagination. Coverage matters more than count: a suite spanning your intents, edge cases, and past incidents beats a large homogeneous one. Then grow it continuously from production triage, which is where the durable value comes from.

What sampling rate should online judges use?

There is no universal number; it depends on traffic volume and budget. A practical pattern is a low flat rate across all traffic for trend detection, 100 percent judging of traces with negative user feedback, and temporarily boosted sampling for new prompt versions or experiment arms. Tune until the judge spend is a small, defensible fraction of total inference cost.

Which model should I use as an LLM judge?

For high-volume online scoring with tight rubrics, a small fast model like claude-haiku-4-5 keeps costs sane. For nuanced offline pairwise comparisons, use a stronger model such as claude-opus-4-8. In every case, pin the judge model version, keep judge prompts in version control, and calibrate against human labels before trusting the scores.

Which tools support both offline and online LLM evaluation?

Langfuse, LangSmith, Braintrust, Arize Phoenix, and W&B Weave all cover tracing, online scoring, and dataset-based offline runs. Promptfoo and DeepEval are strongest for offline CI-style testing, and Ragas provides RAG-specific metrics you can run in either mode. All of them layer over the same architecture described here, so the tool choice is rarely the hard part.

Do offline eval scores predict online performance?

Directionally at best, and only if the offline dataset resembles production traffic. A fresh hand-written suite usually correlates poorly. The correlation improves as you feed production failures back into the golden set, which is the strongest argument for running the closed loop rather than treating offline and online evaluation as separate programs.