Human-in-the-Loop LLM Evaluation

Human in the loop evaluation means putting real people in the scoring path for your LLM outputs instead of trusting a metric or a model alone. You use humans to judge a sampled slice of production traffic, encode their judgment into a rubric, and then use that labeled data to calibrate cheaper automated checks. This article shows the whole loop end to end: sampling, rubric design, an annotation schema, inter-rater agreement, calibrating an LLM judge against human labels, and wiring the result into CI so a bad model change gets blocked before users see it.

If you have ever shipped a prompt change that looked fine in three manual spot checks and then quietly broke a fifth of your traffic, human in the loop evaluation is the discipline that catches it. The point is not to have humans grade everything forever. The point is to spend human attention where it buys you the most signal, then automate the rest with something you have proven agrees with those humans.

Why human in the loop evaluation beats metrics alone

Automated metrics for generative systems are weak on their own. BLEU and ROUGE measure token overlap, which barely correlates with whether an answer is correct or useful. Exact-match works only when there is one canonical string. Embedding similarity tells you two texts are about the same topic, not that one is right and the other is wrong. For anything open ended (summaries, chat replies, code explanations, extraction with judgment calls) the ground truth lives in a human's head, and you have to get it out.

The trap engineers fall into is treating a model as its own judge without ever checking that judge against people. An LLM-as-judge can be excellent, but only after you have measured how often it agrees with human raters on your task. Skip that step and you are optimizing against a number nobody has validated. Human in the loop evaluation is the validation step. It gives you a small, trustworthy set of human labels that everything else gets measured against.

Three things humans catch that metrics miss:

• Subtle factual errors that read fluently. A confidently wrong date or name sails past every overlap metric.
• Instruction violations. The output ignored a constraint ("answer in one sentence", "do not mention pricing") while still scoring well on similarity.
• Tone and safety issues that are context dependent and cannot be regexed.

The loop, start to finish

Here is the shape of a human in the loop evaluation pipeline. Every stage feeds the next.

1. Capture production traffic (inputs, outputs, metadata) into a store you can query.
2. Sample a representative slice, with deliberate oversampling of risky segments.
3. Define a rubric: the exact questions a human answers about each output.
4. Collect labels from at least two annotators per item.
5. Measure inter-rater agreement and resolve disagreements.
6. Calibrate an automated judge (an LLM or a classifier) against the human labels.
7. Run the calibrated judge on all traffic and in CI; route only low-confidence cases back to humans.

Steps 1 through 5 are pure human in the loop work. Steps 6 and 7 are how you scale it so you are not paying humans to read every generation.

Sampling: where human attention goes

You cannot label everything, so sampling decides what your humans even see. Uniform random sampling is the honest baseline: it gives you an unbiased estimate of overall quality. But uniform sampling wastes budget on the easy middle of your distribution. Stratified sampling spends more human time on the parts that matter.

A practical strategy: split traffic into strata (by feature, user segment, input length, or a cheap risk heuristic), then sample within each. Oversample the risky strata and record the sampling weight so you can reweight back to a true population estimate later.

import random
from collections import defaultdict

def stratified_sample(records, key_fn, per_stratum, seed=0):
    rng = random.Random(seed)
    buckets = defaultdict(list)
    for r in records:
        buckets[key_fn(r)].append(r)

    sampled = []
    for stratum, items in buckets.items():
        rng.shuffle(items)
        take = items[:per_stratum]
        weight = len(items) / max(len(take), 1)
        for it in take:
            it["_stratum"] = stratum
            it["_weight"] = weight  # for reweighting to population later
            sampled.append(it)
    return sampled

# Example: oversample long inputs and a high-risk feature
def key_fn(r):
    long = len(r["input"]) > 2000
    return (r["feature"], "long" if long else "short")

batch = stratified_sample(traffic, key_fn, per_stratum=40)

Two rules that save you pain. First, freeze the seed so a sample is reproducible and you can re-pull the exact set. Second, keep the sampling weight attached to every record. When you report "92 percent pass rate" you want it reweighted to the real traffic mix, not the mix your oversampling created.

For a first pass, 150 to 300 labeled items is usually enough to see whether a change moved quality. You do not need thousands to detect a real regression.

Writing a rubric humans actually agree on

The rubric is the heart of human in the loop evaluation, and it is where most teams go wrong. A vague rubric ("rate quality 1 to 5") produces labels that disagree constantly because every annotator invents their own scale. Your job is to remove the guesswork.

Principles that raise agreement:

• Prefer binary or few-level ordinal questions over a 1 to 10 slider. Humans agree on "did it answer the question, yes or no" far more than on "is this a 6 or a 7".
• Decompose. Instead of one "quality" score, ask three concrete questions: is it factually correct, does it follow the instructions, is the tone appropriate. Composite scores hide which dimension broke.
• Give each level a definition and an example. "Score 0: contains a claim that is false or unsupported. Score 1: all claims are supported by the source." Then paste a real example next to each.
• Write a tie-breaker rule for the ambiguous middle so two people resolve it the same way.

A rubric encoded as data, not prose, so your tooling can validate it:

RUBRIC = {
    "factual": {
        "question": "Is every factual claim supported by the provided source?",
        "levels": {
            0: "At least one claim is false or unsupported.",
            1: "All claims are supported.",
        },
    },
    "instructions": {
        "question": "Did the output follow all explicit instructions?",
        "levels": {
            0: "Violated one or more instructions.",
            1: "Followed all instructions.",
        },
    },
    "tone": {
        "question": "Is the tone appropriate for a support reply?",
        "levels": {
            0: "Rude, dismissive, or off-brand.",
            1: "Neutral to helpful.",
        },
    },
}

def validate_label(label):
    for dim, spec in RUBRIC.items():
        if dim not in label:
            raise ValueError(f"missing dimension: {dim}")
        if label[dim] not in spec["levels"]:
            raise ValueError(f"bad value for {dim}: {label[dim]}")
    return True

Pilot the rubric on 20 items with two annotators before you scale. If they disagree on more than a couple, the rubric is ambiguous, not the annotators. Fix the definitions and pilot again. This half hour saves you from throwing out a whole labeling batch.

Collecting labels without building a labeling app

You do not need a full annotation platform to start. A structured spreadsheet or a tiny local form is enough for the first few hundred items. What matters is the schema: one row per (item, annotator) pair, with the raw dimension scores, a free-text note, and timing.

import csv, json, time

def write_tasks(items, path):
    with open(path, "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["item_id", "input", "output", "source",
                    "factual", "instructions", "tone", "note"])
        for it in items:
            w.writerow([it["id"], it["input"], it["output"],
                        it.get("source", ""), "", "", "", ""])

def load_labels(path, annotator):
    rows = []
    with open(path) as f:
        for r in csv.DictReader(f):
            if r["factual"] == "":
                continue  # unlabeled
            rows.append({
                "item_id": r["item_id"],
                "annotator": annotator,
                "factual": int(r["factual"]),
                "instructions": int(r["instructions"]),
                "tone": int(r["tone"]),
                "note": r["note"],
                "ts": time.time(),
            })
    return rows

Assign every item to at least two annotators. The overlap is what lets you measure agreement, and the disagreements are the most informative items in the whole batch: they are exactly the edge cases your rubric or your model handles badly. Blind the annotators to which model version produced an output, otherwise expectations leak into scores.

Measuring agreement so you can trust the labels

Raw percent agreement lies to you. If 95 percent of outputs are fine, two annotators who both rubber-stamp everything will agree 95 percent of the time while contributing zero signal. Cohen's kappa corrects for agreement you would expect by chance. Use it per dimension.

def cohens_kappa(a, b):
    # a, b: equal-length lists of categorical labels from two annotators
    n = len(a)
    cats = sorted(set(a) | set(b))
    idx = {c: i for i, c in enumerate(cats)}
    k = len(cats)
    m = [[0] * k for _ in range(k)]
    for x, y in zip(a, b):
        m[idx[x]][idx[y]] += 1

    observed = sum(m[i][i] for i in range(k)) / n
    row = [sum(m[i]) / n for i in range(k)]
    col = [sum(m[i][j] for i in range(k)) / n for j in range(k)]
    expected = sum(row[i] * col[i] for i in range(k))
    if expected == 1:
        return 1.0
    return (observed - expected) / (1 - expected)

Rough reading of kappa: below 0.2 is noise, 0.2 to 0.4 is weak, 0.4 to 0.6 is moderate, 0.6 to 0.8 is substantial, above 0.8 is strong. If a dimension comes back below 0.4, do not average the scores and move on. The rubric for that dimension is broken. Rewrite the definitions, add examples pulled from the disagreements, and re-label. Agreement is a property of your rubric far more than of your people.

For ordinal scales with more than two levels, weighted kappa (which penalizes far-apart disagreements more than adjacent ones) is a better fit, but the binary version above is the right place to start.

Once agreement is acceptable, resolve the remaining disagreements. A third adjudicator, or a quick discussion, produces the gold label. That adjudicated set is your ground truth for everything downstream.

Calibrating an LLM judge against human labels

Now you scale. You have a few hundred gold-labeled items. Build an LLM-as-judge that answers the same rubric questions, then measure how often it agrees with your humans. Only ship the judge if that agreement is close to the agreement your humans had with each other. A judge that matches human-human kappa is as good as adding another annotator, and it runs on all your traffic for pennies.

Give the judge the same rubric text your humans used, force structured output, and ask for a short reason so you can audit it.

import json

JUDGE_PROMPT = """You are grading an AI output against a rubric.

Rubric:
- factual (0 or 1): {factual}
- instructions (0 or 1): {instructions}
- tone (0 or 1): {tone}

Input given to the AI:
{input}

Source material:
{source}

AI output to grade:
{output}

Return ONLY JSON: {{"factual": 0|1, "instructions": 0|1,
"tone": 0|1, "reason": "one sentence"}}"""

def judge(client, item, rubric):
    prompt = JUDGE_PROMPT.format(
        factual=rubric["factual"]["question"],
        instructions=rubric["instructions"]["question"],
        tone=rubric["tone"]["question"],
        input=item["input"], source=item.get("source", ""),
        output=item["output"],
    )
    resp = client.generate(prompt, temperature=0)  # your SDK call
    return json.loads(resp)

Run the judge over the gold set and compute kappa between the judge and the adjudicated human labels, per dimension:

def judge_vs_human(gold, judge_labels, dim):
    h = [g[dim] for g in gold]
    j = [judge_labels[g["item_id"]][dim] for g in gold]
    return cohens_kappa(h, j)

Interpretation is the whole game. If judge-human kappa on "factual" is 0.7 and your human-human kappa was 0.72, the judge is trustworthy on that dimension. Ship it. If judge-human kappa is 0.3 while humans agreed at 0.75, the judge is unreliable on that dimension no matter how confident it sounds. Keep that dimension human-only, or improve the judge prompt (add few-shot examples drawn from the gold set, or split the question further) and re-measure.

Practical judge notes that move agreement:

• Use temperature 0 for reproducibility.
• Put the rubric definitions and one worked example per level directly in the prompt.
• Grade one dimension per call if a combined call is noisy; it costs more but agreement often jumps.
• Watch for position bias in pairwise "which is better" judging: swap the order and average, or the judge favors whichever answer came first.
• Re-run calibration whenever you change the judge model. A new model version is a new judge and its agreement is not inherited.

Closing the loop: judge in CI, humans on the hard cases

Once you have a calibrated judge, human in the loop evaluation stops being a batch project and becomes continuous. The judge scores all production traffic. It also runs in CI against a frozen eval set so a prompt or model change that drops the factual pass rate fails the build before it merges.

def gate(results, thresholds):
    # results: list of judge outputs on the frozen eval set
    n = len(results)
    report = {}
    ok = True
    for dim, floor in thresholds.items():
        rate = sum(r[dim] for r in results) / n
        report[dim] = round(rate, 3)
        if rate < floor:
            ok = False
    return ok, report

passed, report = gate(judge_results,
                       {"factual": 0.95, "instructions": 0.90, "tone": 0.98})
if not passed:
    raise SystemExit(f"eval gate failed: {report}")

Humans stay in the loop in three targeted ways, none of which require reading every output:

• Low-confidence routing. When the judge is uncertain or two judge runs disagree, send that item to a human. This is active learning: humans see exactly the cases the judge cannot handle.
• Drift audits. Every week, re-label a fresh uniform sample by hand and recompute judge-human kappa. If agreement decays, your traffic has shifted and the judge needs recalibration.
• Regression review. When the CI gate fails, a human reads the failing items to confirm the regression is real before anyone reverts.

That is the sustainable shape: humans define and periodically re-validate the standard, the calibrated judge enforces it at scale, and human attention flows only to the ambiguous edge. You are not replacing human judgment. You are compounding it.

Common failure modes

• Grading with the same model that generated the output, then acting surprised when it grades itself generously. Use a validated judge, and validate it against humans first.
• One annotator per item. With no overlap you cannot measure agreement, so you have no idea whether your labels mean anything.
• Reporting a pass rate off oversampled data without reweighting. Your dashboard says 96 percent while real traffic sits at 88.
• Never re-piloting a rubric after low kappa. Averaging noisy labels does not produce signal, it produces confident noise.
• Freezing the eval set once and never refreshing it. It goes stale, the model overfits to it, and the gate stops protecting real users.
• Trusting judge scores you have never audited. Always keep the judge's one-line reason and read a handful.

FAQ

How many human labels do I actually need? For detecting whether a change moved quality, 150 to 300 gold-labeled items per evaluation is usually enough to see a real shift. For calibrating an LLM judge you want a similar few hundred with two-annotator overlap so kappa is stable. You scale by labeling more only when you need tighter confidence intervals on a specific segment.

What is a good inter-rater agreement to aim for? Cohen's kappa above roughly 0.6 per dimension means your rubric is solid enough to trust. Below 0.4 the rubric is ambiguous and you should rewrite definitions and re-pilot before labeling more. Do not chase 1.0: some genuine ambiguity is normal, and forcing perfect agreement usually means you oversimplified the task.

Can an LLM judge fully replace human evaluation? No, and that is not the goal. A calibrated judge replaces humans only on dimensions where you have proven it agrees with them, and only until traffic drifts. Humans still set the rubric, adjudicate disagreements, handle low-confidence cases, and re-validate the judge on a schedule. Treat the judge as an amplifier of human judgment, not a substitute for it.

Binary labels or a 1 to 5 scale? Start binary or three-level ordinal. Coarse scales produce far higher agreement because there is less room to invent a personal interpretation. If you genuinely need finer resolution, decompose into more binary questions rather than stretching one question across ten levels. You can always aggregate binary dimensions into a composite score later.

How do I keep the eval set from going stale? Refresh it on a schedule. Keep a frozen set for CI comparability, but every few weeks pull a new uniform sample, hand-label it, and recompute judge-human agreement. Rotate a fraction of the frozen set with new production cases so the model cannot overfit to a static benchmark. Drift in judge-human kappa is your signal that recalibration is due.

Where does LLM-as-judge fit versus human in the loop evaluation? They are two stages of one pipeline. Human in the loop evaluation produces the trustworthy labels and the rubric. LLM-as-judge is how you scale enforcement of that rubric cheaply once you have measured its agreement with the humans. Neither works well alone: a judge without human calibration optimizes an unvalidated number, and humans without a judge cannot cover production volume.