Common LLM Eval Mistakes Teams Make Early On

Why Your Eval Setup Is Probably Lying to You

A team ships an LLM feature, everyone tries it in the demo, it feels magical, and the code goes to production. Three weeks later support tickets pile up: the chatbot hallucinated a refund policy, the summarizer dropped a critical clause, the classifier is inconsistent between Monday and Friday. Someone asks "didn't we test this?" and the answer is usually "yes, we ran it on a few examples and it looked good."

That's the pattern behind almost every LLM eval failure we see teams hit early on. Not a lack of effort — a lack of rigor in how "looked good" gets measured. Evaluation for LLM systems is genuinely different from evaluation for traditional software, and teams that treat it like unit testing, or treat it like a vague vibe check, both end up burned, just in different ways.

This post walks through the mistakes we see most often when teams start evaluating LLM applications — RAG pipelines, agents, classifiers, summarizers, whatever the use case — and what to do instead. None of this is theoretical; these are the same failure modes that show up in production incident reviews over and over.

Mistake 1: Eyeballing Outputs Instead of Building a Test Set

The single most common early mistake is having no structured evaluation set at all. The workflow looks like this: write a prompt, run it against three or four inputs pulled from your own head, read the outputs, decide it "feels right," ship it. This works fine for a demo. It falls apart the moment real users send inputs you didn't imagine.

The fix isn't complicated, it's just unglamorous: build a labeled eval set before you optimize the prompt or pick a model. That means collecting real or realistic inputs, writing down what a correct or acceptable output looks like for each one, and treating that set as a regression suite you re-run every time you touch the prompt, the model, or the retrieval logic.

A minimal eval set for a support-ticket classifier might look like this:

eval_set = [
    {
        "input": "My payment failed twice but I was still charged both times.",
        "expected_label": "billing_issue",
    },
    {
        "input": "How do I export my data before canceling?",
        "expected_label": "account_management",
    },
    {
        "input": "The app crashes every time I try to upload a PDF over 10MB.",
        "expected_label": "bug_report",
    },
    {
        "input": "I was charged but the app also crashed during checkout.",
        "expected_label": "billing_issue",  # ambiguous on purpose
    },
]

Notice the last example. A good eval set deliberately includes ambiguous, adversarial, and edge cases — not just the clean examples that make your system look good. Teams that only test happy paths get a false sense of security that evaporates in production.

Start with 30-50 examples if that's all you have time for. It's not statistically rigorous, but it's infinitely better than zero, and it forces you to define "correct" in writing instead of in your head. Pull those examples from real production logs where you have them, since actual user phrasing is weirder and more useful than anything invented at a whiteboard, from domain experts who know where the system tends to fail in ways engineers won't anticipate, and from deliberately constructed adversarial cases — near-duplicate categories, conflicting signals, edge-of-context-window lengths.

One more detail teams get wrong: they write the eval set once and let a single person "eyeball approve" the expected outputs without a second reviewer. If your labels are wrong, your eval actively misleads you — it will flag a correct model output as a failure, and you'll tune a working prompt into a worse one chasing a phantom bug. Have someone else spot-check the labels before trusting the set as a baseline.

Mistake 2: Confusing "It Ran Without an Error" With "It Worked"

LLM outputs almost always look plausible. That's the core danger. A hallucinated API endpoint, a fabricated citation, a confidently wrong summary — none of these throw an exception. The system returns 200 OK with a beautifully formatted, completely wrong answer.

Teams coming from traditional software engineering instinctively reach for the tools they know: does the function return without throwing, does the JSON parse, does the status code look right. Those checks matter, but they only catch a tiny fraction of LLM failure modes. The much larger category — factual correctness, faithfulness to source documents, tone, completeness, instruction-following — requires evaluation methods traditional testing doesn't have.

If you're building a RAG system, "no error" tells you the retrieval call succeeded and the model produced text. It tells you nothing about whether that text is grounded in the retrieved documents. You need a separate check for that:

def check_faithfulness(answer: str, retrieved_chunks: list[str], judge_fn) -> dict:
    """
    Uses a judge model to verify every claim in `answer`
    is supported by the retrieved context.
    """
    context = "\n---\n".join(retrieved_chunks)
    verdict = judge_fn(
        answer=answer,
        context=context,
        instruction=(
            "List each factual claim in the answer. For each claim, "
            "mark it SUPPORTED, CONTRADICTED, or UNSUPPORTED based only "
            "on the provided context. Do not use outside knowledge."
        ),
    )
    return verdict

The point isn't this exact function — it's the mindset shift. "It ran" and "it's correct" are entirely different claims, and conflating them is how ungrounded answers slip through a code review that only checked for crashes.

This mistake is especially sneaky because the failure modes that don't throw errors are often the ones with the highest business cost. A support bot that occasionally 500s gets noticed and fixed fast — it's loud and shows up in error dashboards immediately. A support bot that confidently invents a return policy that doesn't exist looks completely healthy in every infrastructure metric you're tracking. Uptime is 100%, latency is fine, and the only place the failure shows up is in the actual content of the response — precisely what traditional monitoring wasn't built to inspect.

Mistake 3: Picking a Single Metric and Trusting It Blindly

Once teams do start measuring quality, the next trap is reaching for one number and treating it as ground truth. BLEU and ROUGE scores for generated text, cosine similarity between embeddings, exact-match accuracy for classification — these are useful signals, but each one measures a narrow slice of quality and can be gamed or simply miss what matters.

A summarizer can score well on ROUGE while omitting the one sentence that actually mattered to the user, because ROUGE rewards word overlap, not information priority. A RAG answer can have high embedding similarity to a reference answer while stating the opposite conclusion, because similarity captures topic, not polarity or correctness.

The practical fix is to evaluate on multiple axes and refuse to compress them into a single score too early:

• Correctness — is the factual content right, checked against ground truth or source documents
• Faithfulness/groundedness — for RAG or summarization, does the output stay within what the source material supports
• Completeness — did it cover the required points, not just avoid saying wrong things
• Format compliance — does it match the schema, length, or structure the downstream system expects
• Safety/tone — does it avoid the things you specifically don't want, like over-promising or off-brand language
• Latency and cost — quality that arrives ten seconds late or costs ten times your budget isn't shippable quality

Track these separately. A dashboard with five numbers is more useful than a single composite score because it tells you where to intervene when things regress. There's a subtler version of this mistake too: picking the right dimensions but the wrong measurement method for each one. Exact-match string comparison is a bad way to check correctness for open-ended generation, because two answers can be worded completely differently and both be correct. The rule of thumb worth internalizing: use exact-match or schema validation only for genuinely structured outputs, and use a rubric-based judge for qualities like tone or completeness where there's no single correct string, only better and worse answers along a spectrum.

Mistake 4: Never Re-Running Evals After a Prompt or Model Change

This is the mistake that turns a working system into a flaky one. A team tunes a prompt, gets it working well on the cases they were staring at, ships it. Two weeks later, someone tweaks the system prompt to fix an unrelated issue, or the team upgrades to a newer model version, or a library update changes the default temperature. Nobody re-runs the eval set because there wasn't a habit of treating it as a gate.

LLM behavior is not additive the way most code changes are. Adding an instruction to "always cite your sources" can silently degrade performance on a completely different task the same prompt handles, because you've changed the token budget or introduced conflicting instructions. Small wording changes can produce disproportionate output changes — that's true of the underlying models, and pretending your prompt is exempt from it is how regressions sneak in.

Treat your eval set exactly like a test suite in CI:

def run_eval_suite(model_fn, eval_set, threshold=0.85):
    results = []
    for case in eval_set:
        output = model_fn(case["input"])
        score = score_output(output, case["expected_label"])
        results.append({"input": case["input"], "score": score, "output": output})

    pass_rate = sum(r["score"] >= 1 for r in results) / len(results)
    if pass_rate < threshold:
        failures = [r for r in results if r["score"] < 1]
        raise AssertionError(
            f"Eval pass rate {pass_rate:.2%} below threshold {threshold:.2%}. "
            f"{len(failures)} failures — inspect before merging."
        )
    return pass_rate

Wire this into your CI pipeline, or at minimum into your PR checklist, so that any prompt edit, model swap, or retrieval change has to clear the same bar before it merges. The cost of running this is a few dollars in API calls. The cost of not running it is a production regression nobody notices until a user complains.

Model upgrades deserve special caution here, because teams often assume "newer model, same prompt, strictly better results" and skip re-evaluation entirely. That assumption is wrong often enough that it shouldn't be an assumption at all. A newer model can follow instructions more literally, which breaks a prompt that was quietly relying on the old model's tendency to infer intent, or it can be better on average while being worse on your specific long-tail cases, because "better on average" is measured on someone else's benchmark, not yours. Every model swap should re-run the full eval set before it touches production traffic, compared side by side against the previous model's scores on the same cases.

Mistake 5: Testing the Model in Isolation, Not the Full Pipeline

Most real LLM applications are not "prompt in, text out." They're pipelines: retrieval, reranking, prompt assembly, the model call, output parsing, maybe a second model call for verification, then formatting for the UI. Teams frequently eval only the model call — testing the prompt in a playground — and never test the pipeline as a whole.

This misses an enormous class of bugs. Retrieval can return the wrong chunks even when the prompt is perfect. A parser can silently truncate a valid model output because of a regex that doesn't handle a new format the model started producing. A reranker can push the correct document below the context window cutoff. None of these are "prompt problems," and none of them show up if your eval only calls the model directly with hand-picked context.

Evaluate end to end, from the actual user-facing input to the actual user-facing output, going through every component the request would really pass through in production:

def eval_full_pipeline(query: str, expected_facts: list[str]) -> dict:
    retrieved = retriever.search(query, top_k=5)
    reranked = reranker.rerank(query, retrieved)
    prompt = build_prompt(query, reranked)
    raw_output = llm.generate(prompt)
    parsed = parse_response(raw_output)  # the step that silently breaks

    missing = [fact for fact in expected_facts if fact not in parsed["answer"]]
    return {
        "query": query,
        "parsed_ok": parsed is not None,
        "missing_facts": missing,
        "retrieved_doc_ids": [d.id for d in reranked],
    }

If you only test the llm.generate step, you'll ship a beautifully tuned prompt sitting on top of a retrieval bug that's been silently degrading answer quality for a month. This also means your eval infrastructure has to run against the same code path production traffic runs against, not a simplified copy of it. It's tempting to build a separate "eval harness" script that calls the model directly with hand-crafted context and never connect it back to the real retriever and parser. That divergence compounds: someone fixes a bug in the production parser but forgets the eval harness has its own copy, and months later a passing eval run tells you almost nothing about what will happen in production. Whenever possible, the harness should import and call the actual production functions rather than reimplementing lightweight versions for convenience.

Mistake 6: No Error Analysis, Only Aggregate Scores

Even teams that do build eval sets often stop at the aggregate number: "we're at 82% pass rate." That number tells you almost nothing actionable. Is the failure concentrated in one category, one input length, one edge case type? Or is it spread randomly, suggesting a fundamental capability gap versus a fixable systemic issue?

Skipping error analysis means you optimize blind. Teams end up re-writing prompts based on gut feel about what might be wrong, re-running the eval, and being surprised when the score doesn't move — because they never actually looked at which examples failed and why. Instead, after every eval run, group the failures and read them:

• Cluster failures by input characteristics (length, topic, language, presence of numbers or dates)
• Read every failed case manually if the set is small enough — don't just look at the score
• Categorize the failure mode: wrong retrieval, hallucination, format violation, refusal, incomplete answer, wrong tone
• Fix the highest-frequency category first, not the most interesting one

This is the difference between "the eval score went from 82% to 85%, we're not sure why" and "we found that 60% of failures were on multi-part questions where the model only answered the first part, so we added an explicit checklist instruction and fixed most of that category." The second is how real improvement happens. A useful habit here is keeping a running failure log as a shared document rather than letting failed cases disappear back into the eval script's output — record the input, the actual output, the expected output, and a one-line hypothesis for the root cause on every failure. Over a few weeks this log becomes one of the most valuable artifacts your team has, because patterns invisible in any single eval run become obvious once you can scroll through fifty of them at once.

Mistake 7: Treating LLM-as-a-Judge as a Magic Oracle

As teams scale past manual review, using a second LLM to grade the first LLM's outputs becomes necessary — you can't have a human read 10,000 outputs every time you change a prompt. But teams often treat LLM-as-a-Judge as infallible the moment they set it up, skipping the validation step that makes it trustworthy in the first place.

A judge model has its own biases: it tends to favor longer, more verbose answers regardless of correctness; it can be inconsistent across runs if temperature isn't controlled; it can inherit the same blind spots as the model being judged if they're the same underlying model family; and a vague judge prompt produces vague, unreliable verdicts. The fix is to validate your judge before trusting it, and to be specific in how you ask it to grade:

JUDGE_PROMPT = """
You are evaluating whether an AI assistant's answer is correct and complete
given the reference answer below. Do not reward length or style — grade
only factual correctness and completeness.

Question: {question}
Reference answer: {reference}
Assistant's answer: {candidate}

Respond with a JSON object:
{{
  "verdict": "correct" | "partially_correct" | "incorrect",
  "reasoning": "<one sentence explaining the verdict>"
}}
"""

def judge_output(question, reference, candidate, judge_model):
    prompt = JUDGE_PROMPT.format(
        question=question, reference=reference, candidate=candidate
    )
    return judge_model.generate(prompt, temperature=0)

Before relying on this in your pipeline, sample 30-50 of the judge's verdicts and have a human check agreement. If the judge and human disagree more than roughly 10-15% of the time, the judge prompt needs work before you trust it as a gate. Set temperature to 0 for consistency, use a stronger or differently-trained model as the judge where possible, and re-validate the judge whenever you change the underlying task, not just once at setup.

There's also a scope mistake teams make with LLM-as-a-Judge: using one broad, generic judge prompt to grade every kind of task in the system. A judge instruction tuned for grading factual correctness in a RAG answer is a poor fit for grading whether a generated email has the right tone. Each task category benefits from its own judge prompt with criteria specific to what "good" means for it, plus its own small validation sample against human labels — otherwise "rate this response 1-10" means something different for every task type and the judge has no way to know which meaning you intended. It's also worth running the judge continuously against a sample of live production traffic, not just your static eval set, since it can process volume a human reviewer can't and gives you an early warning signal for drift.

Mistake 8: No Plan for Eval Set Drift

An eval set built on day one reflects the inputs you imagined on day one. Real user traffic drifts — new phrasing, new edge cases, new failure modes you didn't anticipate — and a static eval set stops representing reality within a few months. Teams that never refresh it end up "passing" an eval that no longer measures what production actually throws at the system.

Build a lightweight process for this from the start: route a sample of production failures, flagged by users, by the judge, or by low-confidence outputs, into a review queue, and periodically fold the genuinely new failure patterns into the eval set as new labeled cases. This keeps the eval set alive and adversarial instead of a museum piece you stopped trusting six months in.

A cadence that works for most teams: review the failure queue weekly while the product is young and changing fast, add a handful of new cases from real failures each time, and retire old cases that no longer reflect how the product is used. Treat eval set growth as a metric worth watching in its own right — a set that hasn't grown in three months despite active development is a sign nobody's feeding production learnings back into it, and the pass rate you're reporting to stakeholders is quietly becoming less meaningful every week it stays static.

Putting It Together

None of these mistakes are exotic. They're the natural result of treating LLM systems like either deterministic software, testing a few cases and shipping, or like magic, where it looks good so it ships. The teams that avoid production surprises are the ones that build a labeled eval set early, measure multiple quality dimensions instead of one score, gate every prompt or model change behind that eval set, test the full pipeline rather than the model in isolation, actually read their failures instead of just tracking the aggregate number, and validate their LLM-as-a-Judge setup against human judgment before trusting it to scale their review process.

Get those habits in place before the traffic ramps up, and evaluation stops being the thing that catches fires after they start — it becomes the thing that keeps most of them from starting at all.