Evaluating Fine-Tuned Models Against Base Models

The Fine-Tuning Hangover Nobody Warns You About

You spent three days curating a dataset, burned through your compute budget, and finally got your fine-tuned model to converge. The loss curve looks great. Your team is excited. Someone asks the obvious question: "So is it actually better than the base model?" And you realize you don't have a good answer.

This is the most common failure mode in applied LLM work. Teams treat fine-tuning as the finish line when it's really the halfway point. The other half — rigorous, honest evaluation against the base model you started with — is where most of the real engineering judgment lives. Skip it, and you risk shipping a model that's worse than what you had before, just because it "feels" more aligned with your use case during a few manual spot checks.

This article is a practical walkthrough of how to evaluate a fine-tuned model against its base model: what to measure, how to structure the comparison, which traps to avoid, and how to build an evaluation harness you can actually trust. We'll use concrete examples throughout, including code you can adapt directly. By the end, you'll have a repeatable process instead of a vibe check.

Why "It Feels Better" Is Not Evidence

Before getting into methodology, it's worth being blunt about why informal evaluation fails.

When you fine-tune a model on your own data, you develop a strong prior. You've read the training examples. You know what "good" looks like for your domain. When you then test the fine-tuned model on a handful of prompts, you're not evaluating objectively — you're pattern-matching against expectations you built during data curation. This is confirmation bias wearing an engineer's hat.

There are three specific ways this goes wrong in practice:

• Cherry-picked prompts. Engineers tend to test with prompts similar to the training distribution, where the fine-tuned model was always going to shine. The base model never gets a fair fight because the test set wasn't designed to be neutral.
• Recency bias in reading outputs. After staring at training examples for days, a fine-tuned model's output style feels "right" even when the actual content is no better, or worse, than the base model's.
• No regression check. Fine-tuning on a narrow dataset can quietly degrade capabilities the model had before — general reasoning, following unrelated instructions, refusing genuinely harmful requests. If you only test the narrow domain, you won't see the damage until a user hits it in production.

Evaluation exists to remove your own judgment from the loop as much as possible and replace it with something reproducible.

Define What "Better" Means Before You Test

The single biggest mistake in model evaluation is starting to run tests before deciding what you're actually trying to measure. "Better" is not one thing. It decomposes into at least four separate axes, and a fine-tuned model can win on some and lose on others.

1. Task accuracy. Does the model produce the correct answer, classification, or structured output for the specific task you fine-tuned it for? This is usually the easiest to measure because you likely have labeled examples.

2. Style and format adherence. Did fine-tuning succeed at making the model consistently output your desired format — JSON schema, tone, length, structure — more reliably than the base model with prompting alone?

3. Generalization and robustness. Does the model still perform well on inputs that are similar to, but not identical to, the training distribution? Overfitting to narrow patterns is the classic fine-tuning failure.

4. Capability retention. Has the model lost general abilities it had before — multi-step reasoning, following out-of-domain instructions, basic factual recall — as a side effect of the fine-tuning process? This is sometimes called "catastrophic forgetting" and it's underappreciated because it doesn't show up unless you specifically test for it.

Write these four axes down as a rubric before you run a single eval. If you skip this step, you'll end up building an eval suite that only measures axis 1, declare victory, and ship a model that's secretly worse on axis 4.

Build a Held-Out Test Set That Wasn't Used for Anything

This sounds obvious, but it's violated constantly. Your test set needs three properties:

• It was never seen during training, including as a "few-shot example" pulled into a prompt template.
• It was never used to pick hyperparameters or decide when to stop training (that's what a validation set is for — keep them separate).
• It includes both in-distribution examples (similar to training data) and out-of-distribution examples (deliberately different, to catch generalization failures).

A reasonable split for a mid-sized fine-tuning project: 70% train, 15% validation (used during training for early stopping and checkpoint selection), 15% test (touched only once, at the very end, for the base-vs-fine-tuned comparison).

Here's a simple way to structure this with a manifest file, so you always know exactly what was used where:

import json
import random

def split_dataset(examples, seed=42):
    random.seed(seed)
    shuffled = examples[:]
    random.shuffle(shuffled)

    n = len(shuffled)
    train_end = int(n * 0.70)
    val_end = int(n * 0.85)

    splits = {
        "train": shuffled[:train_end],
        "validation": shuffled[train_end:val_end],
        "test": shuffled[val_end:],
    }

    for name, split in splits.items():
        with open(f"{name}.jsonl", "w") as f:
            for ex in split:
                f.write(json.dumps(ex) + "\n")

    manifest = {name: len(split) for name, split in splits.items()}
    with open("split_manifest.json", "w") as f:
        json.dump(manifest, f, indent=2)

    return splits

Keep split_manifest.json under version control. Six months from now, when someone asks "was this example in the training set," you want a definitive answer, not a guess.

Metric Selection: Match the Metric to the Task Shape

Different task types need different metrics. Using the wrong metric is how teams end up with a dashboard full of green numbers that don't actually reflect quality.

Classification or extraction tasks (sentiment labels, entity extraction, intent classification): use precision, recall, F1, and a confusion matrix. Don't just report accuracy — a model that always predicts the majority class can have high accuracy and be useless.

Structured output tasks (JSON generation, function-calling arguments): measure schema validity rate separately from content correctness. A model can produce perfectly valid JSON with wrong values, or invalid JSON with correct values buried inside. These are different failure modes requiring different fixes.

Open-ended generation tasks (summarization, long-form answers, chat responses): this is where you need a combination of automated proxy metrics (ROUGE, BLEU, or embedding similarity for a rough signal) and human or LLM-based judgment for actual quality, since n-gram overlap metrics correlate poorly with real usefulness.

Retrieval-augmented tasks: measure faithfulness to the retrieved context separately from overall answer quality. A fine-tuned model might get better at sounding confident while getting worse at actually grounding its answer in the provided documents — a dangerous combination.

Here's a minimal harness that runs both models against the same test set and computes exact-match and F1 for a structured extraction task:

from dataclasses import dataclass
from typing import Callable

@dataclass
class EvalResult:
    model_name: str
    exact_match: float
    f1: float
    schema_valid_rate: float
    n_examples: int

def token_f1(pred: str, gold: str) -> float:
    pred_tokens = pred.lower().split()
    gold_tokens = gold.lower().split()
    if not pred_tokens or not gold_tokens:
        return 0.0
    common = set(pred_tokens) & set(gold_tokens)
    if not common:
        return 0.0
    precision = len(common) / len(pred_tokens)
    recall = len(common) / len(gold_tokens)
    return 2 * precision * recall / (precision + recall)

def evaluate_model(
    model_name: str,
    generate_fn: Callable[[str], str],
    validate_schema_fn: Callable[[str], bool],
    test_set: list[dict],
) -> EvalResult:
    exact_matches = 0
    f1_scores = []
    valid_schema_count = 0

    for example in test_set:
        prediction = generate_fn(example["input"])
        gold = example["output"]

        if prediction.strip() == gold.strip():
            exact_matches += 1
        f1_scores.append(token_f1(prediction, gold))

        if validate_schema_fn(prediction):
            valid_schema_count += 1

    n = len(test_set)
    return EvalResult(
        model_name=model_name,
        exact_match=exact_matches / n,
        f1=sum(f1_scores) / n,
        schema_valid_rate=valid_schema_count / n,
        n_examples=n,
    )

Run this once with the base model's generate_fn and once with the fine-tuned model's, on the exact same test_set, and put the two EvalResult objects side by side. That side-by-side comparison — not either number in isolation — is the actual deliverable.

The Capability Retention Check

This is the step teams skip most often, and it's the one that catches production incidents before they happen.

Take a small, general-purpose benchmark that has nothing to do with your fine-tuning domain — a subset of a reasoning benchmark, a set of instruction-following prompts, or even a handful of hand-written "sanity check" prompts covering basic arithmetic, simple logic, and format-following unrelated to your task. Run both models against it.

If the fine-tuned model's score on this unrelated benchmark drops meaningfully compared to the base model, you have evidence of catastrophic forgetting, even if your domain-specific metrics improved. This is a real trade-off you need to make consciously, not one you want to discover from a confused user report three weeks after launch.

A practical rule of thumb: if fine-tuning improves your target metric by 15 points but drops general instruction-following by 20 points, you likely need to adjust your training approach — lower learning rate, fewer epochs, more diverse training data, or techniques like LoRA with a smaller rank to constrain how much the weights can shift away from the base model's original behavior.

Where LLM-as-a-Judge Fits In

For open-ended outputs, exact-match and F1 metrics only get you so far. Two responses can use completely different wording and both be excellent, or both be technically similar and both be wrong in the same subtle way. This is where LLM-as-a-Judge earns its place in the pipeline: using a strong, separate model to score or compare outputs against a rubric, standing in for a human rater at a fraction of the cost and turnaround time.

The key discipline is to make the judge as rigorous as any other part of your eval: give it a clear rubric, show it the input and both candidate outputs, ask for structured scoring rather than a vague preference, and — critically — randomize which output is labeled "A" and which is "B" for every comparison. Judge models have measurable position bias; if the fine-tuned model's output is always shown first, you'll get an inflated win rate that has nothing to do with actual quality.

JUDGE_PROMPT = """You are evaluating two AI responses to the same user query.
Score each response from 1-5 on:
- Correctness: factually and logically sound
- Completeness: fully addresses the query
- Format: matches the requested output format

User query: {query}

Response A: {response_a}

Response B: {response_b}

Return JSON only:
{{"response_a": {{"correctness": int, "completeness": int, "format": int}},
  "response_b": {{"correctness": int, "completeness": int, "format": int}},
  "reasoning": "one sentence"}}
"""

def judge_pair(query, base_output, finetuned_output, judge_fn, randomize=True):
    import random
    if randomize and random.random() < 0.5:
        a, b, swapped = finetuned_output, base_output, True
    else:
        a, b, swapped = base_output, finetuned_output, False

    prompt = JUDGE_PROMPT.format(query=query, response_a=a, response_b=b)
    result = judge_fn(prompt)

    if swapped:
        result["response_a"], result["response_b"] = result["response_b"], result["response_a"]

    return result

Aggregate these pairwise judgments across your whole test set into a win rate, tie rate, and loss rate for the fine-tuned model versus the base model. A fine-tuned model that wins 55% of comparisons, ties 30%, and loses 15% tells a very different story than one that wins 90% — the first suggests a modest, real improvement; the second should make you suspicious of judge bias or a leaked training example, and is worth spot-checking by hand.

One caution: don't treat the judge's verdict as ground truth on its own. Use it as one signal alongside your automated metrics and the capability retention check. When the judge disagrees sharply with your F1 scores, that disagreement is informative — it might mean your reference answers are too rigid, or it might mean the judge is being fooled by confident-sounding but wrong text.

Cost, Latency, and the Metrics Nobody Puts on the Eval Dashboard

A fine-tuned model that's 3 points better on accuracy but twice as slow, or that requires a much larger base checkpoint to fine-tune effectively, might not be a net win for your product. Evaluation isn't complete until you've measured the operational side too:

• Latency per request, measured under realistic load, not a single warm request in a notebook.
• Cost per 1,000 requests, factoring in whether the fine-tuned model lets you drop down to a smaller base model and still hit your quality bar — this is often the actual business case for fine-tuning in the first place.
• Output length distribution. Fine-tuned models sometimes learn to be verbose or overly terse as an artifact of training data statistics rather than genuine quality improvement. Compare average token counts between base and fine-tuned outputs on the same prompts.

Put these next to your quality metrics in the same report. A 4% accuracy gain that comes with a 40% latency increase is a trade-off someone with product context needs to sign off on, not something an eval script should silently decide.

Assembling the Comparison Report

All the pieces above are only useful if they end up in one place someone can actually read and act on. A comparison report that decision-makers will trust needs, at minimum:

• The four-axis rubric (task accuracy, style adherence, generalization, capability retention) with a score or pass/fail per axis, for both models.
• The held-out test set size and composition, so reviewers can judge statistical significance rather than eyeballing two numbers.
• Automated metric tables (F1, exact match, schema validity) side by side.
• LLM-as-a-Judge win/tie/loss rates with the randomization method disclosed.
• Cost and latency numbers under comparable load.
• A handful of representative examples — including at least one where the fine-tuned model performs worse — so readers see the trade-offs rather than a highlight reel.

That last point matters more than it sounds. Every comparison report tends to accumulate a "greatest hits" section of the fine-tuned model's best outputs. Force yourself to include at least one clear regression example. It builds trust in the rest of the report and it's usually where the next round of training data curation should focus.

Common Pitfalls That Invalidate an Otherwise Good Eval

A few mistakes show up repeatedly across teams doing this for the first time, and each one is enough to make your conclusions wrong even if everything else was done carefully.

• Testing at a different temperature or sampling setting for each model. If the base model runs at temperature 0.7 and the fine-tuned model at temperature 0, you're not comparing the models — you're comparing sampling strategies. Fix generation parameters identically across both, or explicitly test the effect of temperature as its own variable.
• Different prompt templates for base vs. fine-tuned. It's tempting to give the base model a more elaborate prompt with few-shot examples to "give it a fair chance," but that changes what you're measuring. If you want to isolate the effect of fine-tuning, hold the prompt constant and let the fine-tuned model's advantage come from its weights, not from extra scaffolding you only gave one side.
• Small test sets driving big claims. A test set of 20 examples cannot support a claim like "12% improvement in accuracy" with any confidence — the noise band on a sample that small is often larger than the reported gain. Use enough examples that a simple confidence interval or bootstrap resample gives you a believable margin of error, and report that margin alongside the point estimate.
• Ignoring failure clustering. Averages hide a lot. Break down errors by category or difficulty bucket. A fine-tuned model might be dramatically better on easy cases and no better — or worse — on the hard 10% that actually matter to your product.
• Re-using the validation set as the test set. If checkpoint selection during training was based on validation loss, and you then report your final comparison numbers on that same validation set, you've contaminated your own result. The model was implicitly tuned to do well there.

A Minimal Checklist Before You Ship

Before declaring a fine-tuned model production-ready, walk through this list:

1. Held-out test set confirmed untouched by training or checkpoint selection.
2. Task accuracy metrics computed identically for base and fine-tuned models, same prompt template, same sampling parameters.
3. Capability retention checked against an out-of-domain benchmark or sanity-check prompt set.
4. LLM-as-a-Judge comparison run with randomized ordering and a clear rubric.
5. Cost and latency measured under realistic load, not a single local run.
6. At least one regression example documented, not just wins.
7. Confidence interval or sample-size justification included for headline numbers.

If you can check all seven boxes, you have a defensible answer the next time someone asks "is it actually better?" If you can't, that's your to-do list, not a reason to ship anyway and hope nobody asks.

Closing Thoughts

Fine-tuning is the easy part to get excited about — it's tangible, it produces a new checkpoint, and the loss curve gives you a satisfying sense of progress. Evaluation is less glamorous, but it's the part that actually determines whether you shipped an improvement or a regression wearing a better costume. Treat your held-out test set with the same discipline as your training data, measure across all four axes instead of just the one that flatters your fine-tune, and lean on LLM-as-a-Judge for the open-ended cases where automated metrics run out of signal. Do that consistently, and "is it better?" stops being a question you dread and becomes one you can answer in a single glance at your comparison report.