Evaluating a Fine-Tuned Model

Fine-tuning evaluation is the process of measuring whether a fine-tuned model is genuinely better than the model you started from, on the task you care about, without secretly getting worse at everything else. It is not a single accuracy number. A real fine-tuning evaluation compares the tuned model against the base model on a held-out set the model never saw during training, checks for regressions on general capability, and looks at cost and latency before you ship. This article walks through the full loop with runnable Python and real commands, so you can tell the difference between a model that learned and a model that just memorized your training data.

Why fine-tuning evaluation is different from normal ML eval

In classic supervised learning you have a clean test split, a fixed label, and a metric like F1. Fine-tuning a language model breaks a few of those assumptions, which is why fine-tuning evaluation needs its own playbook.

First, the output is usually free text, not a class label. Two answers can both be correct while sharing almost no tokens, so exact-match and BLEU-style overlap metrics undercount real quality. Second, the base model already knows a lot. Your fine-tune is a delta on top of a very capable starting point, so the interesting question is not "is it good" but "is it better than the base, and is the improvement worth the added serving cost." Third, fine-tuning can quietly damage capabilities you did not test. A model tuned to write terse SQL can forget how to hold a normal conversation. This is regression, and catching it is half the job of fine-tuning evaluation.

Because of all this, a trustworthy fine-tuning evaluation always has three legs:

• A task metric on a held-out set that reflects the job you tuned for.
• A regression check on general capability so you notice collateral damage.
• An operational read on cost, latency, and failure modes before rollout.

Skip any leg and you are guessing.

Build the held-out evaluation set first

The single biggest mistake in fine-tuning evaluation is measuring on data that leaked into training. If an example appears in both your training file and your eval file, the model can memorize it, and your score becomes a lie. Build and freeze the eval set before you train.

A practical rule: split by source entity, not by row. If your data is customer support tickets, hold out entire customers, not random tickets, because tickets from the same customer share phrasing and templates that leak signal. Same idea for documents, users, or time periods.

Here is a splitter that partitions by a group key and then checks for near-duplicate leakage between the two sides using a cheap normalized hash.

import json, hashlib, random
from collections import defaultdict

def normalize(text):
    return " ".join(text.lower().split())

def group_split(rows, group_key, eval_frac=0.15, seed=13):
    groups = defaultdict(list)
    for r in rows:
        groups[r[group_key]].append(r)
    keys = list(groups)
    random.Random(seed).shuffle(keys)
    n_eval = int(len(keys) * eval_frac)
    eval_keys = set(keys[:n_eval])
    train = [r for k in keys if k not in eval_keys for r in groups[k]]
    held = [r for k in eval_keys for r in groups[k]]
    return train, held

def leakage_report(train, held, field="prompt"):
    train_hashes = {hashlib.sha1(normalize(r[field]).encode()).hexdigest() for r in train}
    collisions = [r for r in held if hashlib.sha1(normalize(r[field]).encode()).hexdigest() in train_hashes]
    return len(collisions)

rows = [json.loads(l) for l in open("all_data.jsonl")]
train, held = group_split(rows, group_key="customer_id")
print("train", len(train), "held", len(held), "leaks", leakage_report(train, held))

If leaks is anything but zero, fix it before training. Exact-normalized collision is the floor, not the ceiling. For higher confidence run an embedding similarity pass and drop held-out rows whose nearest training neighbor is above a threshold like 0.95 cosine.

Aim for at least a few hundred held-out examples. Under about 100 your metric has so much variance that a two-point swing tells you nothing. If your task has natural slices (language, ticket type, difficulty), keep enough per slice to read them separately, because an average can hide a slice that got much worse.

Pick metrics that match the task

The right metric for a fine-tuning evaluation depends on how constrained the output is. Match the metric to the shape of the answer.

For closed outputs, where there is one correct answer or a small set (classification, extraction, routing, structured JSON), use exact match or field-level accuracy. These are cheap, deterministic, and hard to game. If you tuned for JSON output, validate against a schema and count a response as wrong if it fails to parse, no partial credit.

import json
from jsonschema import validate, ValidationError

schema = {
    "type": "object",
    "required": ["intent", "priority"],
    "properties": {
        "intent": {"type": "string"},
        "priority": {"enum": ["low", "medium", "high"]},
    },
}

def score_structured(prediction_text, gold):
    try:
        pred = json.loads(prediction_text)
        validate(pred, schema)
    except (json.JSONDecodeError, ValidationError):
        return 0.0
    return 1.0 if pred == gold else 0.0

For open outputs, where many phrasings are correct (summaries, replies, explanations), token-overlap metrics are weak. You have two honest options: task-specific programmatic checks, or an LLM judge. Programmatic checks are best when correctness has verifiable structure. If the task is generating a shell command, run it in a sandbox and check the exit code and output. If it is code, run the unit tests. If it is a citation-grounded answer, verify every cited fact appears in the source. These checks beat any judge because they measure the thing you actually want.

When there is no programmatic oracle, an LLM judge is the pragmatic default in 2026, and the next section covers how to use one without fooling yourself.

Score open-ended answers with an LLM judge

An LLM judge is a strong model prompted to score another model's output against a rubric. It scales far past human review and correlates well with human preference when set up carefully. It also fails in specific, known ways, so treat the judge as an instrument you must calibrate, not an oracle.

Use these guardrails in every LLM-judge fine-tuning evaluation:

• Score against a reference answer and an explicit rubric, not vibes. Give the judge the input, the gold reference, and the candidate, and ask for a score on named criteria.
• Force a structured verdict with reasoning before the score, so you can audit disagreements.
• Randomize or hide which model produced which answer to fight position and identity bias. Judges tend to favor the first option shown and answers that look like their own style.
• Calibrate the judge against a few dozen human labels. If judge and humans disagree badly, fix the rubric before trusting the judge at scale.

Here is a compact judge harness. It is provider-agnostic in shape, calling a chat function you implement against whatever model you use as the judge.

import json, random

RUBRIC = """You are grading a support reply.
Score each criterion 1-5 and return strict JSON:
{"correctness": int, "completeness": int, "tone": int, "reasoning": str}
correctness: factually right vs the reference.
completeness: covers what the reference covers.
tone: professional and on-brand."""

def judge(chat, question, reference, candidate):
    msg = (f"{RUBRIC}\n\nCUSTOMER:\n{question}\n\n"
           f"REFERENCE:\n{reference}\n\nCANDIDATE:\n{candidate}")
    raw = chat(system="Return only JSON.", user=msg, temperature=0)
    return json.loads(raw)

def run_pairwise(chat, question, reference, answer_a, answer_b):
    # hide identity: shuffle order, ask which is better
    items = [("A", answer_a), ("B", answer_b)]
    random.shuffle(items)
    labeled = "\n\n".join(f"OPTION {name}:\n{txt}" for name, txt in items)
    prompt = (f"Question:\n{question}\n\nReference:\n{reference}\n\n{labeled}\n\n"
              'Which option is better? Return {"winner":"A"|"B","why":str}.')
    verdict = json.loads(chat(system="Return only JSON.", user=prompt, temperature=0))
    winner_slot = verdict["winner"]
    return dict(items)[winner_slot]  # maps back to real answer text

Run the judge at temperature=0 for repeatability, and run each comparison twice with the order flipped. If the winner changes when you swap positions, mark it a tie. A judge that flips on order is telling you the two answers are effectively equal, which is useful information on its own.

Always compare against the base model and a frozen baseline

A fine-tuning evaluation with only one number is meaningless. The tuned score has to be read against the base model on the exact same held-out set, scored by the exact same harness. Run three configurations through the identical pipeline:

• The base model, no fine-tuning, with your best prompt.
• The base model with a few in-context examples, if few-shot is a realistic alternative to tuning.
• The fine-tuned model.

If your fine-tune does not clearly beat a well-prompted base and a few-shot base, you may not need to fine-tune at all, and that is a legitimate outcome to discover early. Prompting is cheaper to maintain than a tuned checkpoint.

Wrap the whole thing so every model runs the same code path. The only variable that changes is the model handle.

def evaluate(model_fn, held, judge_chat):
    scores = []
    for ex in held:
        pred = model_fn(ex["prompt"])
        v = judge(judge_chat, ex["prompt"], ex["reference"], pred)
        composite = (v["correctness"] * 0.5
                     + v["completeness"] * 0.3
                     + v["tone"] * 0.2)
        scores.append(composite)
    return sum(scores) / len(scores), scores

base_avg, base_scores = evaluate(call_base, held, judge_chat)
tuned_avg, tuned_scores = evaluate(call_tuned, held, judge_chat)
print(f"base {base_avg:.3f}  tuned {tuned_avg:.3f}  delta {tuned_avg - base_avg:+.3f}")

Freeze this baseline. Every future tuning run compares against the same held-out set and the same base numbers, so you can see whether run 7 actually beat run 3 or just got lucky.

Is the improvement real? Check significance and slices

A raw delta like +0.12 average score feels convincing until you realize your eval set has 80 examples and the variance is huge. Before you celebrate, ask whether the difference would survive a different sample. Since you have per-example scores for both models on the same inputs, a paired bootstrap is the honest, assumption-light test.

import random

def paired_bootstrap(base_scores, tuned_scores, n=10000, seed=1):
    rng = random.Random(seed)
    diffs = [t - b for b, t in zip(base_scores, tuned_scores)]
    k = len(diffs)
    wins = 0
    for _ in range(n):
        sample = [diffs[rng.randrange(k)] for _ in range(k)]
        if sum(sample) / k > 0:
            wins += 1
    return wins / n

p_better = paired_bootstrap(base_scores, tuned_scores)
print(f"probability tuned truly beats base: {p_better:.3f}")

If that probability is around 0.95 or higher, the improvement is likely real. If it is 0.7, you have a coin flip dressed up as a win. Collect more eval data or accept that the gain is not established.

Then break the score down by slice. Averages lie. A model can gain three points overall while losing ten points on your hardest ticket type, and if that hard type is your highest-value traffic, the average was hiding a disaster.

from collections import defaultdict

def by_slice(held, scores, key):
    buckets = defaultdict(list)
    for ex, s in zip(held, scores):
        buckets[ex[key]].append(s)
    return {k: sum(v) / len(v) for k, v in buckets.items()}

print("tuned by type:", by_slice(held, tuned_scores, "ticket_type"))
print("base by type:", by_slice(held, base_scores, "ticket_type"))

Read the two dictionaries side by side. Any slice where tuned is below base is a regression that needs a decision before you ship.

Guard against regressions and overfitting

Fine-tuning evaluation is not only about the target task. It is about what the model lost. Two failure modes deserve dedicated checks.

Overfitting shows up as a model that scores well on eval but has clearly memorized. Sniff for it by comparing performance on held-out examples that are near-duplicates of training data versus those that are genuinely novel. If the model aces the near-duplicates and stumbles on the novel ones, it memorized rather than generalized. This is exactly why the leakage check earlier matters, and why you keep a small "hard novel" slice that resembles nothing in training.

Capability regression shows up as damage to general behavior. Keep a small standing regression set of general prompts, a few dozen is enough, covering plain conversation, refusal on unsafe requests, instruction following, and basic reasoning. Run it against base and tuned every time. If the tuned model starts producing malformed output on a normal chat prompt, or loosens a safety refusal it used to hold, that is a blocker regardless of how good the task score looks.

regression_prompts = [json.loads(l) for l in open("regression_set.jsonl")]

def regression_check(model_fn, judge_chat, threshold=3.5):
    failures = []
    for ex in regression_prompts:
        pred = model_fn(ex["prompt"])
        v = judge(judge_chat, ex["prompt"], ex["reference"], pred)
        if v["correctness"] < threshold:
            failures.append((ex["prompt"][:60], v["correctness"]))
    return failures

fails = regression_check(call_tuned, judge_chat)
print(f"{len(fails)} regression failures")
for p, s in fails:
    print(f"  [{s}] {p}")

Treat any new regression failure as ship-blocking until a human looks at it. A model that is two points better at your task and noticeably worse at holding a safe, coherent conversation is usually a net loss.

Measure cost, latency, and failure modes before rollout

The last leg of fine-tuning evaluation is operational, and teams skip it constantly. A tuned model that is marginally better but noticeably slower or more expensive per call may not clear the bar once you multiply by production volume.

Measure three things on the held-out run:

• Latency distribution, especially the p95 and p99 tail, not just the average. Tail latency is what users feel.
• Token cost per request, input plus output, since a tuned model that writes longer answers can quietly raise your bill even at the same per-token rate.
• Failure rate: fraction of responses that fail schema validation, time out, or trip a safety filter.

import time

def profile(model_fn, held):
    latencies, out_tokens, failures = [], [], 0
    for ex in held:
        t0 = time.perf_counter()
        try:
            pred = model_fn(ex["prompt"])
        except Exception:
            failures += 1
            continue
        latencies.append(time.perf_counter() - t0)
        out_tokens.append(len(pred.split()))  # rough proxy; use real token counts
    latencies.sort()
    p95 = latencies[int(len(latencies) * 0.95)]
    return {
        "p50_s": latencies[len(latencies) // 2],
        "p95_s": p95,
        "avg_out_tokens": sum(out_tokens) / len(out_tokens),
        "failures": failures,
    }

print(profile(call_tuned, held))

Put the quality delta and the operational delta in one table in your head: a fine-tune that is clearly better on quality, holds all regressions, and costs about the same is an easy ship. One that is marginally better on quality but slower and pricier is a judgment call that belongs to whoever owns the budget.

A repeatable fine-tuning evaluation checklist

Run the same sequence for every tuning run so results stay comparable across weeks and models.

1. Freeze a held-out set split by entity, and confirm zero leakage against training.
2. Pick metrics that match output shape: exact match or schema validation for closed outputs, programmatic oracle or calibrated LLM judge for open outputs.
3. Score base, few-shot base, and tuned through the identical harness on the identical set.
4. Run a paired bootstrap to confirm the delta is real, not sampling noise.
5. Break scores down by slice and flag any slice where tuned lost to base.
6. Run the standing regression set for capability and safety; treat new failures as blockers.
7. Profile latency tail, token cost, and failure rate.
8. Record every number, seed, prompt, and model handle so the run reproduces.

The reproducibility step is doing more work than it looks. Store the eval set hash, the judge prompt, the judge model name, and the random seeds next to the scores. Six weeks later, when someone asks why run 3 looked better than run 9, that record is the only thing that lets you answer instead of shrug.

FAQ

How many examples do I need in a fine-tuning evaluation set? Enough that a small real improvement is not drowned by noise. A few hundred held-out examples is a reasonable floor for a single-task eval, and you want enough per slice to read slices separately. Under about 100 the variance is large enough that a paired bootstrap will usually tell you the delta is not established, which is your signal to collect more.

Can I trust an LLM judge to score my fine-tune? Yes, if you calibrate it. Score against a reference and an explicit rubric, force reasoning before the score, hide which model produced which answer, run at temperature zero, and check agreement against a few dozen human labels before trusting it at scale. Judges have known biases toward the first option shown and toward their own writing style, and the setup above is what neutralizes them. Where a programmatic oracle exists, such as running code or checking a schema, prefer it over any judge.

What is the difference between evaluating on training loss and evaluating the fine-tuned model? Training and validation loss tell you the optimization converged and did not overfit the token distribution. They do not tell you the model is good at your task. Loss going down is necessary but not sufficient. A fine-tuning evaluation runs the actual model on held-out inputs and scores the outputs the way your product cares about, which is the only measurement that predicts production behavior.

How do I catch capability regressions from fine-tuning? Keep a small standing regression set of general prompts covering conversation, instruction following, safety refusals, and basic reasoning, and run base versus tuned on it every time. Fine-tuning narrowly can degrade broad capability, and the average task score will hide it. Any new failure on that set is a blocker until a human reviews it.

Should I fine-tune at all, or just prompt better? Find out empirically before you invest. Include a well-prompted base model and a few-shot base model as baselines in your evaluation. If the fine-tune does not clearly and significantly beat both, prompting is cheaper to build and maintain, and discovering that early saves you an entire training and serving pipeline. Fine-tuning earns its keep when you need consistent format, a specialized style, lower per-call token cost from shorter prompts, or behavior that few-shot cannot reliably produce.

How often should I re-run the fine-tuning evaluation? Every tuning run, against the same frozen held-out set and the same baseline numbers, so runs are comparable. Also re-run when the base model you tune from changes version, when your data distribution shifts, or before any production rollout. Treat the eval harness as standing infrastructure, not a one-time gate.