DeepEval for Fine-Tuned Model Comparison

The Problem With "It Feels Better" Evaluation

You just finished a fine-tuning run. You fed it a few hundred examples, watched the loss curve dip, and now you're staring at a checkpoint file wondering if it's actually better than the base model. Most teams answer this question by typing ten prompts into a notebook and eyeballing the outputs. That approach works right up until it doesn't — usually the moment a subtly regressed model ships to production and support tickets start piling up.

Fine-tuning changes a model's behavior in ways that are easy to miss with manual spot-checks. A model can get noticeably better at your target task while quietly getting worse at instruction-following, or more prone to hallucination, or less consistent in tone. Loss curves and perplexity scores tell you the model learned *something*, but they don't tell you if it learned the *right* thing, and they definitely don't tell you how it stacks up against the model you're trying to replace.

This is where DeepEval earns its keep. DeepEval is an open-source LLM evaluation framework built around the idea that model outputs should be tested the same way you test code — with assertions, metrics, and a test suite that runs in CI. Instead of "does this look right to me," you get "does this pass a G-Eval correctness check with a threshold of 0.7 across 200 held-out examples, and how does that compare to the previous checkpoint on the exact same examples."

In this article, we'll build a complete, repeatable comparison harness for fine-tuned models using DeepEval. We'll cover dataset construction, metric selection, running parallel evaluations across checkpoints, statistical comparison, and wiring the whole thing into a CI pipeline so every new fine-tune gets judged before it gets deployed.

Why Comparison Is Different From Single-Model Evaluation

Evaluating one model in isolation and comparing two models against each other are related but distinct problems. When you evaluate a single model, you're asking "is this good enough to ship." When you compare models, you're asking "is Model B meaningfully better than Model A, on the same inputs, under the same conditions."

That second question has traps that single-model evaluation doesn't:

• Test set leakage across runs. If your baseline was evaluated on one sample of your dataset and the fine-tuned model on another, any difference you observe could just be dataset variance, not model improvement.
• Metric drift between evaluation runs. If you use an LLM-as-judge metric like G-Eval, the judge model itself has some run-to-run variance. You need to control for that or you'll chase noise.
• Aggregate scores hiding regressions. A fine-tuned model can improve average score while regressing badly on a specific subcategory (e.g., getting better at summarization but worse at refusing unsafe requests). Comparison needs to be sliced, not just averaged.
• Cost and latency tradeoffs. A model that scores 2% higher but costs 3x more tokens per response, or is twice as slow, might not be "better" for your use case even if the quality metric says so.

DeepEval addresses these by letting you define a single EvaluationDataset, run it against multiple models with the same metrics, and compare results test-case-by-test-case rather than just as aggregate numbers. That's the structural difference that makes the comparison trustworthy.

There's a related trap worth naming separately: correlated checkpoints. When you're comparing checkpoint 800 against checkpoint 1200 from the same training run, the two models often agree on the easy 80% of your dataset and differ only on the hard remainder. If your dataset is mostly easy examples, the aggregate delta between checkpoints will look tiny even when the underlying behavior change is real and meaningful on the inputs that matter. This is a strong argument for deliberately over-representing hard and edge-case examples in your comparison set relative to how often they occur in production traffic — you're not trying to estimate production accuracy here, you're trying to maximize the harness's ability to tell two similar models apart.

Setting Up the Comparison Harness

Start with the DeepEval installation and a shared evaluation dataset. The key design decision here is that the dataset must be identical across every model you test — same inputs, same expected outputs, same retrieval context if you're doing RAG evaluation.

pip install deepeval
deepeval login  # optional, only if using Confident AI for dashboards

Now define the dataset as a reusable object rather than inline in your test function. This is the single most important habit for fair comparison — build it once, freeze it, reuse it everywhere.

# dataset.py
from deepeval.dataset import EvaluationDataset
from deepeval.test_case import LLMTestCase

golden_examples = [
    {
        "input": "Summarize the refund policy for orders placed over 30 days ago.",
        "expected_output": "Orders placed over 30 days ago are not eligible for a refund, but store credit may be issued at the discretion of support staff.",
    },
    {
        "input": "A customer asks why their subscription renewed despite cancelling. Explain the likely cause.",
        "expected_output": "The cancellation likely occurred after the renewal date cutoff, so the current billing cycle still processed before cancellation took effect.",
    },
    # ... 150-300 more examples pulled from your held-out validation split
]

def build_dataset(model_generate_fn, model_name: str) -> EvaluationDataset:
    test_cases = []
    for ex in golden_examples:
        actual_output = model_generate_fn(ex["input"])
        test_cases.append(
            LLMTestCase(
                input=ex["input"],
                actual_output=actual_output,
                expected_output=ex["expected_output"],
                additional_metadata={"model": model_name},
            )
        )
    return EvaluationDataset(test_cases=test_cases)

Notice that additional_metadata tags each test case with the model name. This becomes essential later when you're slicing results — DeepEval doesn't force you to keep separate dataframes per model if you tag consistently.

The model_generate_fn parameter is deliberately abstracted. Whether you're calling a local checkpoint via transformers, a hosted fine-tune via an API, or a LoRA adapter loaded on top of a base model, the harness doesn't care. It just needs a function that takes a prompt string and returns a completion string.

Choosing Metrics That Actually Detect Fine-Tuning Regressions

Picking the right metrics matters more than picking a lot of metrics. For fine-tuned model comparison, you generally want a mix of reference-based metrics (compare against expected output) and reference-free metrics (judge quality independent of a gold answer).

G-Eval is the workhorse for this. It uses an LLM judge with a custom rubric, and you can tune the rubric to match exactly what your fine-tune was supposed to improve.

# metrics.py
from deepeval.metrics import GEval, AnswerRelevancyMetric, HallucinationMetric
from deepeval.test_case import LLMTestCaseParams

correctness_metric = GEval(
    name="Correctness",
    criteria="Determine whether the actual output is factually consistent with the expected output and directly answers the input question.",
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
        LLMTestCaseParams.EXPECTED_OUTPUT,
    ],
    threshold=0.7,
)

tone_metric = GEval(
    name="Tone Adherence",
    criteria="Determine whether the actual output matches a professional, concise support-agent tone without being curt or overly verbose.",
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
    threshold=0.7,
)

relevancy_metric = AnswerRelevancyMetric(threshold=0.75)
hallucination_metric = HallucinationMetric(threshold=0.5)

comparison_metrics = [correctness_metric, tone_metric, relevancy_metric, hallucination_metric]

A few practical notes from actually running this pattern:

• Write the G-Eval criteria around the specific behavior you fine-tuned for. If you fine-tuned to improve tone, a generic "correctness" metric won't detect that improvement. If you fine-tuned to reduce hallucination, make sure HallucinationMetric is in your suite, not just AnswerRelevancyMetric.
• Keep the judge model fixed across all comparisons. If you evaluate the baseline with GPT-4o as judge and the fine-tune with a different judge model weeks later, your comparison is invalid. Pin the judge model version in your harness config.
• Use at least one deterministic metric alongside the LLM-judged ones. Something like exact match rate or a ROUGE-based metric for structured outputs gives you a sanity check that isn't subject to judge variance.
• Don't just reuse the metric suite from your last project. A metric suite tuned for a summarization fine-tune won't transfer cleanly to a tool-calling fine-tune. Rewrite the G-Eval criteria text for the actual task, even if it feels like extra setup work — a generic rubric produces generic, low-signal scores.
• Set thresholds relative to your current baseline, not an arbitrary round number. If your base model scores 0.55 on tone adherence today, a threshold of 0.9 will fail every checkpoint you ever test and tell you nothing about relative improvement. Anchor thresholds to where you actually are, then move them up over time as checkpoints clear the bar.

It's worth having a short combined metric alongside the individual ones, especially once you're tracking four or five metrics per checkpoint. A weighted composite score gives you a single number for quick triage, while the underlying per-metric breakdown stays available for the deeper read.

def composite_score(metric_means: dict, weights: dict) -> float:
    return sum(metric_means[name] * weight for name, weight in weights.items())

weights = {
    "Correctness": 0.4,
    "Tone Adherence": 0.25,
    "Answer Relevancy": 0.2,
    "Hallucination": 0.15,
}

Treat the composite score as a triage tool, not the final verdict. Two checkpoints with the same composite score can have very different risk profiles — one might be balanced across all four metrics, the other might be excellent on tone and mediocre on hallucination. Always look at the breakdown before promoting anything based on the composite alone.

Running the Same Suite Against Multiple Checkpoints

Here's the core comparison loop. The pattern is: load each model, generate outputs over the identical dataset, evaluate with the identical metric set, and store results keyed by model name.

# run_comparison.py
from deepeval import evaluate
from deepeval.evaluate import AsyncConfig
from dataset import build_dataset, golden_examples
from metrics import comparison_metrics

def load_baseline_model():
    # e.g. base instruct model via your inference client
    from inference_client import BaselineClient
    client = BaselineClient(model="base-instruct-v1")
    return client.generate

def load_finetuned_model(checkpoint_path: str):
    from inference_client import FineTunedClient
    client = FineTunedClient(checkpoint_path=checkpoint_path)
    return client.generate

def run_for_model(model_generate_fn, model_name: str):
    dataset = build_dataset(model_generate_fn, model_name)
    results = evaluate(
        test_cases=dataset.test_cases,
        metrics=comparison_metrics,
        async_config=AsyncConfig(run_async=True, max_concurrent=10),
    )
    return results

if __name__ == "__main__":
    baseline_results = run_for_model(load_baseline_model(), "base-instruct-v1")
    finetuned_results = run_for_model(
        load_finetuned_model("./checkpoints/ft-run-042"), "ft-run-042"
    )

Two details in this snippet matter for correctness, not just style. First, async_config with bounded concurrency keeps evaluation fast without hammering your judge model's rate limits — with 200+ test cases times four metrics, you're issuing close to a thousand judge calls, and unbounded concurrency will get you throttled. Second, the models are loaded and generated against the dataset *inside* run_for_model rather than sharing generation across models, which guarantees each model sees the exact same input list with no cross-contamination.

Aggregating and Comparing Results Programmatically

evaluate() returns per-test-case metric scores, not just a pass/fail summary. Pull those into a structure you can diff directly, rather than reading them off a terminal printout.

# compare_results.py
import statistics

def summarize(results, model_name: str):
    summary = {}
    for metric_name in ["Correctness", "Tone Adherence", "Answer Relevancy", "Hallucination"]:
        scores = [
            mr.score
            for tc in results.test_results
            for mr in tc.metrics_data
            if mr.name == metric_name
        ]
        summary[metric_name] = {
            "mean": statistics.mean(scores),
            "stdev": statistics.stdev(scores) if len(scores) > 1 else 0.0,
            "pass_rate": sum(1 for s in scores if s >= 0.7) / len(scores),
        }
    return summary

def diff_summaries(baseline_summary, finetuned_summary):
    print(f"{'Metric':<20}{'Baseline':<12}{'Fine-Tuned':<12}{'Delta':<10}")
    for metric_name in baseline_summary:
        base_mean = baseline_summary[metric_name]["mean"]
        ft_mean = finetuned_summary[metric_name]["mean"]
        delta = ft_mean - base_mean
        flag = "REGRESSION" if delta < -0.03 else ""
        print(f"{metric_name:<20}{base_mean:<12.3f}{ft_mean:<12.3f}{delta:+.3f}  {flag}")

The -0.03 threshold for flagging a regression is a judgment call, not a DeepEval default — pick a number that reflects how much noise you've observed in repeated runs of the same model. If you rerun the baseline twice and see 0.02 swings from judge variance alone, anything smaller than that is not a real signal.

This is also where per-test-case diffing pays off. Aggregate means can look flat while individual test cases flip badly in both directions. Add a pass that surfaces the worst regressions by test case, not just by metric average:

def find_regressions(baseline_results, finetuned_results, metric_name="Correctness"):
    base_by_input = {tc.input: tc for tc in baseline_results.test_results}
    regressions = []
    for tc in finetuned_results.test_results:
        base_tc = base_by_input.get(tc.input)
        if not base_tc:
            continue
        base_score = next(m.score for m in base_tc.metrics_data if m.name == metric_name)
        ft_score = next(m.score for m in tc.metrics_data if m.name == metric_name)
        if ft_score < base_score - 0.15:
            regressions.append({
                "input": tc.input,
                "baseline_score": base_score,
                "finetuned_score": ft_score,
                "finetuned_output": tc.actual_output,
            })
    return sorted(regressions, key=lambda r: r["finetuned_score"] - r["baseline_score"])

Reading through the top ten regressions by hand usually reveals a pattern — a specific input category, a length issue, a format the fine-tune stopped respecting — much faster than staring at an aggregate score.

Checking Whether the Delta Is Real or Just Noise

Before you declare a winner, ask whether the difference you're seeing would survive a second run. LLM-as-judge metrics carry inherent variance because the judge itself is a probabilistic model, and that variance compounds across a few hundred test cases. A checkpoint that scores 0.61 against a baseline's 0.59 is not obviously better — it might just be judge noise.

The cheapest way to check this is to run the same model through the harness twice and measure how much its own scores move between runs, before you compare it against anything else.

def self_consistency_check(model_generate_fn, model_name: str, trials: int = 2):
    trial_means = []
    for _ in range(trials):
        dataset = build_dataset(model_generate_fn, model_name)
        results = evaluate(test_cases=dataset.test_cases, metrics=comparison_metrics)
        scores = [
            mr.score
            for tc in results.test_results
            for mr in tc.metrics_data
            if mr.name == "Correctness"
        ]
        trial_means.append(statistics.mean(scores))
    return {
        "trial_means": trial_means,
        "spread": max(trial_means) - min(trial_means),
    }

If the spread from running the exact same model twice is larger than the delta you observed between baseline and fine-tune, you don't have a real result — you have noise, and no amount of narrative confidence will fix that. In practice, the fix is rarely "run it more times." A bigger, more representative golden dataset reduces variance far more efficiently than repeating the same small dataset over and over, because a larger sample size shrinks the judge's aggregate variance directly. Treat a suspiciously small delta as a signal to grow your dataset before you trust the comparison.

Slicing Comparisons by Category

Averages across your whole test set hide the interesting story. A model fine-tuned on customer support transcripts might improve dramatically on billing questions while quietly regressing on technical troubleshooting questions, and that split is invisible in an overall mean.

Tag your golden dataset with categories up front, then slice the comparison along those tags.

categorized_examples = [
    {"input": "...", "expected_output": "...", "category": "billing"},
    {"input": "...", "expected_output": "...", "category": "technical"},
    {"input": "...", "expected_output": "...", "category": "account_management"},
]

def summarize_by_category(results, examples_with_category, metric_name="Correctness"):
    category_scores = {}
    input_to_category = {ex["input"]: ex["category"] for ex in examples_with_category}
    for tc in results.test_results:
        category = input_to_category.get(tc.input, "uncategorized")
        score = next(m.score for m in tc.metrics_data if m.name == metric_name)
        category_scores.setdefault(category, []).append(score)
    return {cat: statistics.mean(scores) for cat, scores in category_scores.items()}

Run this for both the baseline and the fine-tune, and print the two dictionaries side by side. In practice, this step is what turns "the fine-tune scored 4% higher overall" into an actionable finding like "the fine-tune improved billing accuracy by 12% but technical troubleshooting dropped 8%, so it needs another training pass with more technical examples before it ships."

Wiring This Into CI So Every Checkpoint Gets Judged

A comparison harness that only runs manually gets skipped under deadline pressure. Turn it into a pytest suite that DeepEval can run in CI, gated on a real threshold rather than a human glancing at numbers.

# test_finetune_regression.py
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from metrics import comparison_metrics
from dataset import golden_examples
from inference_client import FineTunedClient

client = FineTunedClient(checkpoint_path="./checkpoints/latest")

@pytest.mark.parametrize("example", golden_examples)
def test_finetune_meets_bar(example):
    actual_output = client.generate(example["input"])
    test_case = LLMTestCase(
        input=example["input"],
        actual_output=actual_output,
        expected_output=example["expected_output"],
    )
    assert_test(test_case, comparison_metrics)

Run it with:

deepeval test run test_finetune_regression.py -n 8

The -n 8 flag parallelizes across workers, which matters once your golden set grows past a hundred examples. Wire this into your training pipeline so that after every fine-tuning job finishes, the new checkpoint automatically runs through this suite before it's promoted to a staging or production alias. A checkpoint that fails assert_test on more than your tolerated regression rate should block promotion, the same way a failing unit test blocks a merge.

If you're using Confident AI alongside DeepEval, each CI run also gets logged with full test-case-level detail, so you get a historical view of every checkpoint's scores over time rather than just the latest pass/fail — useful when someone asks six weeks later why a particular checkpoint got promoted.

Common Pitfalls When Comparing Fine-Tuned Models

A few mistakes show up repeatedly when teams build this kind of harness for the first time:

• Evaluating on training data. If any of your golden examples overlap with the fine-tuning data, the fine-tuned model will look artificially strong. Keep a strict held-out split and verify it before every comparison run, not just once at setup.
• Comparing models with different prompt templates. If your fine-tune expects a different system prompt or chat template than the base model, and you don't account for that in your model_generate_fn, you're measuring prompt engineering differences, not fine-tuning differences.
• Ignoring latency and cost in the "better" verdict. A model that's marginally more correct but meaningfully slower or more expensive per call might be a net loss for your product. Log token counts and latency alongside quality metrics, and report all three together.
• Trusting a single evaluation run. LLM-judge metrics have variance. Run the comparison two or three times and check whether the delta between baseline and fine-tune is larger than the run-to-run noise of either model alone. If it isn't, you don't have a result yet.
• Only testing the happy path. Golden datasets built entirely from clean, well-formed examples won't catch a fine-tune that's become brittle on edge cases, adversarial inputs, or malformed requests. Deliberately include a slice of messy, ambiguous, or edge-case inputs in your golden set.

Bringing It Together

The pattern across this entire harness is consistency: same dataset, same metrics, same judge model, same prompt templates, across every model you evaluate. That consistency is what turns "this new checkpoint feels better" into a defensible, reproducible verdict you can put in front of a team or a stakeholder. DeepEval doesn't do the thinking for you — you still have to pick metrics that reflect what you actually fine-tuned for, build a golden dataset that represents real traffic, and read the per-test-case regressions instead of trusting a single aggregate number. But it gives you the scaffolding to make that thinking repeatable and to catch regressions before your users do.

If you want to go deeper on building evaluation pipelines like this one — including RAG-specific metrics, synthetic dataset generation, and setting up automated evaluation gates in production — check out the DeepEval Tutorial course on teachyou.ai, where we build this exact kind of harness from scratch across a full project.