Building a Custom LLM Judge Rubric From Scratch

Why your first LLM judge rubric is probably wrong

Every team that ships an AI feature eventually hits the same wall: manual review of model outputs doesn't scale, but "just ask GPT to rate it 1-10" produces numbers nobody trusts. You run the same eval twice and get different scores. Your team argues about what a "7" means. Someone ships a regression because the judge gave it an 8.5 and nobody looked closer.

The problem isn't that LLM judges don't work. It's that most rubrics are built backwards — someone writes a vague scoring prompt, throws a temperature-0 call at it, and calls it evaluation. A real rubric is a specification. It defines exactly what "good" means for your task, gives the judge model concrete anchors instead of adjectives, and gets validated against human judgment before it touches your CI pipeline.

This article walks through building a custom LLM judge rubric from first principles: picking a scoring scheme, writing criteria that don't collapse into "vibes," structuring the judge prompt, calibrating against human labels, and wiring the whole thing into a repeatable eval harness. By the end you'll have a rubric template and a working Python judge you can adapt to summarization, RAG answers, agent tool-use, or customer support replies.

Start with the failure modes, not the score

The biggest mistake in rubric design is starting with a number scale. Start instead with a list of ways your system's output can be bad. Pull these from real production logs, support tickets, or a red-team pass — not from imagination.

For a RAG customer-support bot, a realistic failure inventory might look like:

• Answer contradicts the retrieved context (hallucination)
• Answer is technically correct but ignores part of the user's question
• Answer cites a source that doesn't support the claim
• Tone is curt or robotic when the user is clearly frustrated
• Answer is correct but buries the actual answer in three paragraphs of hedging
• Answer recommends an action the user can't actually take (wrong plan tier, deprecated feature)

Once you have this list, group it into 3-6 dimensions. Dimensions are the axes your rubric will score independently — never collapse everything into one "quality" number, because a single scalar hides exactly the failure you're trying to catch. For the support-bot example, you'd land on something like:

• Faithfulness — does the answer only assert what the context supports?
• Completeness — does it address every part of the user's question?
• Actionability — can the user actually do what's suggested?
• Tone — is it appropriate to the user's apparent emotional state?

Four dimensions is a good default. Fewer than three and you're probably merging distinct failure modes; more than six and your judge prompt gets so long that the model starts skimming it, and your human calibration effort multiplies.

Choosing a scoring scheme that doesn't lie to you

There are three common scoring schemes for LLM judges, and picking the wrong one is where most rubrics quietly fail.

1. Likert scales (1-5 or 1-10). These feel natural but are the least reliable in practice. LLMs (like humans) compress toward the middle and struggle to distinguish a 6 from a 7 consistently. If you must use a numeric scale, keep it short — 1 to 4, not 1 to 10 — and write an explicit definition for every single number, not just the endpoints.

2. Binary pass/fail per criterion. Far more reliable. "Does the answer contain a claim not supported by the retrieved context? Yes/No." Binary judgments are easier for the model to apply consistently and easier for you to audit — you can grep for every "No" on faithfulness and read them in five minutes. The tradeoff is you lose granularity; a "pass" on completeness might still be a mediocre answer.

3. Pairwise comparison. Instead of scoring an output in isolation, show the judge two candidate outputs (e.g., current model vs. previous version, or two prompt variants) and ask which is better and why. Pairwise comparisons are the most robust to model idiosyncrasies because the judge doesn't need an absolute reference point — it just needs relative judgment, which LLMs are demonstrably better at.

My default recommendation for a from-scratch rubric: binary pass/fail per dimension, with a structured comment field, plus an optional pairwise mode for A/B testing prompt changes. You get the auditability of binary decisions and can still aggregate a pass-rate metric per dimension for dashboards.

Writing rubric criteria that survive contact with a real model

A rubric criterion is useless if two competent humans reading it would score the same output differently. The fix is to write criteria the way you'd write acceptance criteria for a unit test — concrete, falsifiable, and anchored with examples.

Bad criterion:

Faithfulness: Is the answer faithful to the source material?

This invites the judge to interpret "faithful" however it wants. Better:

Faithfulness: Mark FAIL if the answer contains any claim, number,
policy detail, or recommendation that is not directly stated or
directly inferable from the provided context chunks. Paraphrasing
is fine. Adding outside knowledge, even if factually true in
general, is a FAIL. Mark PASS only if every substantive claim
traces back to the context.

Examples:
- Context says "refunds are processed within 5-7 business days."
  Answer says "you'll get your refund in about a week." -> PASS
  (reasonable paraphrase)
- Context says "refunds are processed within 5-7 business days."
  Answer says "refunds are usually instant for verified accounts."
  -> FAIL (fabricated exception not in context)

Notice the pattern: a one-line rule, an explicit statement of what counts as a violation, and 2-3 worked examples covering the boundary cases, not just the obvious ones. The boundary examples matter most — "obviously good" and "obviously bad" outputs rarely need a judge at all. The rubric earns its keep on the ambiguous middle 20%.

Do this for every dimension. Yes, it's tedious. That tedium is the actual work of building a rubric — the judge prompt engineering afterward is comparatively easy.

Structuring the judge prompt

Once your criteria are written, the judge prompt is mostly plumbing. The structure that has held up best across teams I've worked with:

1. Role and stakes — tell the judge what it's evaluating and why it matters, briefly.
2. Inputs — the original query, the context (if RAG), and the candidate answer, each clearly delimited.
3. Criteria — your dimensions, each with its rule and examples, presented as a numbered list.
4. Output format — force structured output (JSON) so you can parse it deterministically. Never free-text a judge output you plan to aggregate programmatically.
5. Chain-of-thought before verdict — ask for brief reasoning per dimension before the final label. This measurably improves judge accuracy and gives you an audit trail when a verdict looks wrong.

Here's a working version using the Anthropic API with Claude as the judge model:

import json
import anthropic

client = anthropic.Anthropic()

RUBRIC = """
You are evaluating a customer-support bot's answer for a SaaS product.
Score the answer against each dimension below. For each dimension,
think briefly, then give a verdict of PASS or FAIL.

1. FAITHFULNESS
Rule: FAIL if the answer asserts any claim, number, policy detail,
or recommendation not directly stated or directly inferable from
the context. Paraphrasing is fine; adding outside knowledge is FAIL.

2. COMPLETENESS
Rule: FAIL if the answer ignores any distinct sub-question the user
asked. A sub-question is "distinct" if answering it requires
different information than the other parts of the query.

3. ACTIONABILITY
Rule: FAIL if the answer recommends a step the user cannot actually
take given their stated plan/context (e.g. a feature gated to a
higher tier, a deprecated menu path).

4. TONE
Rule: FAIL if the user's message signals frustration (explicit
complaint, exclamation points, repeated contact) and the answer
does not include at least one acknowledging phrase before the
solution.

Respond with ONLY valid JSON in this exact shape:
{
  "faithfulness": {"reasoning": "...", "verdict": "PASS|FAIL"},
  "completeness": {"reasoning": "...", "verdict": "PASS|FAIL"},
  "actionability": {"reasoning": "...", "verdict": "PASS|FAIL"},
  "tone": {"reasoning": "...", "verdict": "PASS|FAIL"}
}
"""

def judge(query: str, context: str, answer: str) -> dict:
    prompt = f"""{RUBRIC}

USER QUERY:
{query}

RETRIEVED CONTEXT:
{context}

CANDIDATE ANSWER:
{answer}
"""
    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        temperature=0,
        messages=[{"role": "user", "content": prompt}],
    )
    raw = response.content[0].text.strip()
    return json.loads(raw)


if __name__ == "__main__":
    result = judge(
        query="My refund hasn't shown up yet and I'm on the Pro plan, can you expedite it?",
        context="Refunds are processed within 5-7 business days after approval. Expedited refunds are not available on any plan.",
        answer="I understand the wait is frustrating. Refunds typically take 5-7 business days to process, and unfortunately expedited refunds aren't available even on the Pro plan. If it's been longer than 7 business days, let me know and I'll escalate it.",
    )
    print(json.dumps(result, indent=2))

Temperature is pinned to 0 for reproducibility, the output is forced JSON, and the reasoning field is generated before the verdict — matching the order in the schema encourages the model to actually reason rather than rationalize a verdict it already committed to.

Calibrating against human judgment

A rubric you haven't validated against humans is a rubric you're guessing about. Before you trust the judge for anything that gates a deploy, run a calibration pass:

1. Sample 30-50 real outputs, weighted toward the ambiguous cases (skip the trivially perfect and trivially broken ones — they don't tell you anything about judge quality).
2. Have 2 human reviewers score them independently against the same rubric, blind to each other's scores and to the judge's scores.
3. Run the LLM judge on the same set.
4. Compute agreement: human-vs-human (your ceiling) and judge-vs-human (your actual number).

If human-vs-human agreement on a dimension is only 70%, don't expect the judge to hit 95% — your rubric criterion is probably still ambiguous, and the fix is to tighten the written rule, not to blame the model. If judge-vs-human agreement is meaningfully below human-vs-human agreement, read the disagreements. You'll usually find one of three things: the rubric wording has a loophole the model is exploiting literally, the judge is missing context it needs (add it to the prompt), or the judge model is just weaker on that particular dimension than expected and needs more examples in the rubric.

A simple agreement check in code:

from collections import Counter

def cohen_kappa(rater_a: list[str], rater_b: list[str]) -> float:
    assert len(rater_a) == len(rater_b)
    n = len(rater_a)
    agree = sum(a == b for a, b in zip(rater_a, rater_b)) / n

    counts_a = Counter(rater_a)
    counts_b = Counter(rater_b)
    labels = set(rater_a) | set(rater_b)
    expected = sum(
        (counts_a[l] / n) * (counts_b[l] / n) for l in labels
    )

    if expected == 1:
        return 1.0
    return (agree - expected) / (1 - expected)

human_labels = ["PASS", "FAIL", "PASS", "PASS", "FAIL"]
judge_labels = ["PASS", "FAIL", "FAIL", "PASS", "FAIL"]
print(f"Kappa: {cohen_kappa(human_labels, judge_labels):.2f}")

Cohen's Kappa above 0.6 is generally considered acceptable agreement; above 0.8 is strong. Anything below 0.4 means the rubric isn't ready to gate anything and needs another revision pass before you trust its verdicts.

Guarding against judge bias

LLM judges carry systematic biases you need to design around, not hope away:

• Position bias in pairwise mode. The judge tends to favor whichever answer is presented first (or second, depending on the model). Fix: run every pairwise comparison twice with the order swapped, and only count a decision as valid if it agrees both times.
• Length bias. Longer answers get rated as more "thorough" even when they're padded with hedging. Counter this explicitly in the rubric: "Length alone is not evidence of completeness. A short, correct answer should PASS; a long answer that restates the question is not more complete."
• Self-preference bias. A judge model tends to rate outputs from the same model family more favorably. If you're evaluating Claude outputs, using a Claude judge is fine for day-to-day iteration, but for a final release gate, cross-check a sample with a different judge model or human review to catch family bias.
• Leniency drift. Judges asked to give holistic scores drift toward generous ratings over long batches. Binary per-criterion scoring reduces this because there's less room for the model to "round up."

None of these biases mean LLM judges are unusable — they mean you build the same kind of defensive checks you'd build around any measurement system with known systematic error.

Wiring the rubric into a repeatable eval harness

A rubric that lives in a notebook doesn't protect you from regressions. The last step is turning it into something that runs automatically against every prompt change or model upgrade.

A minimal harness structure:

import json
from dataclasses import dataclass


@dataclass
class EvalCase:
    id: str
    query: str
    context: str
    answer: str


def run_eval_suite(cases: list[EvalCase]) -> dict:
    results = []
    for case in cases:
        verdicts = judge(case.query, case.context, case.answer)
        results.append({"id": case.id, **verdicts})

    dimensions = ["faithfulness", "completeness", "actionability", "tone"]
    pass_rates = {}
    for dim in dimensions:
        passes = sum(1 for r in results if r[dim]["verdict"] == "PASS")
        pass_rates[dim] = passes / len(results)

    failures = [
        r for r in results
        if any(r[dim]["verdict"] == "FAIL" for dim in dimensions)
    ]

    return {
        "pass_rates": pass_rates,
        "total_cases": len(cases),
        "failing_cases": failures,
    }


if __name__ == "__main__":
    cases = [
        EvalCase(
            id="refund-001",
            query="My refund hasn't shown up, can you expedite it?",
            context="Refunds take 5-7 business days. No expedited option exists.",
            answer="Refunds usually arrive within a week; I don't have a way to speed it up, sorry.",
        ),
    ]
    report = run_eval_suite(cases)
    print(json.dumps(report, indent=2))

Run this suite against a fixed set of eval cases (ideally 50-200, covering your known failure modes plus a random production sample) every time you change a prompt, swap a model version, or update your retrieval pipeline. Track pass rates per dimension over time — a drop in "faithfulness" pass rate after a prompt tweak is a concrete, actionable signal in a way that "the outputs feel a bit off" never is. Store the failing cases alongside the reasoning field the judge produced; that reasoning is often more useful for debugging than the verdict itself, because it tells you where the judge (and often the underlying model) is drawing the wrong line.

Common mistakes that quietly wreck rubrics

A few patterns show up repeatedly in rubrics that don't hold up under scrutiny:

• Testing multiple things in one criterion. "Is the answer correct and well-formatted?" conflates two failure modes into one verdict. Split it — you want to know independently if content is wrong versus if formatting is bad, because the fixes are completely different.
• No negative examples. A rubric with only positive examples teaches the judge what good looks like but not where the line is. Always include at least one near-miss FAIL example per criterion.
• Rubric written once, never revisited. Your rubric should evolve every time you find a production failure the judge missed. Treat rubric criteria like test cases — add one every time a bug slips through.
• Ignoring cost and latency. A four-dimension judge call with chain-of-thought reasoning is not free. If you're running it on every production request rather than in a batch eval, budget for it explicitly, or sample a percentage of live traffic instead of scoring everything.
• Trusting the judge more than the humans who built the rubric. The judge is only as good as the criteria you wrote. When judge and human disagree, the default assumption should be "the rubric wording is ambiguous," not "the human is wrong."

Versioning your rubric like code

Once a rubric is in production, it needs the same discipline you'd apply to a schema migration. Give every rubric a version string and log it alongside every judge verdict — rubric_version: "faithfulness-v3" — so that when pass rates shift, you can tell whether the underlying answers changed or the rubric itself did. Teams that skip this step end up debugging a "regression" for an afternoon before realizing someone tightened the completeness criterion the week before and the drop in pass rate is expected, not a bug.

Keep rubric changes in a plain text or YAML file under version control, not buried in a prompt string inside application code. A diff-able rubric file means a criterion change shows up in code review like any other logic change, and a reviewer who disagrees with a new FAIL condition can push back before it ships. It also means you can re-run an old eval suite against an old rubric version to confirm historical numbers still reproduce — useful when a stakeholder asks "why did our quality score jump 15 points last quarter" and the honest answer is "we loosened the tone criterion," not "the model got better."

A practical layout:

evals/
  rubrics/
    support_bot_v1.yaml
    support_bot_v2.yaml
  cases/
    support_bot_eval_set.jsonl
  results/
    2026-06-01_v1_baseline.json
    2026-07-01_v2_after_prompt_change.json

This also makes rollback trivial. If a new rubric version turns out to be too strict — flagging correct answers as failures because a wording edge case wasn't anticipated — you revert the YAML file, not the judge code.

When to add more judges instead of more criteria

There's a temptation, once a rubric is working, to keep bolting dimensions onto it: faithfulness, completeness, actionability, tone, then formatting, then brand voice, then citation style. Past five or six dimensions, a single judge call starts doing too much in one pass, and accuracy on each individual dimension quietly degrades because the model is juggling more context and more rules simultaneously.

The better move at that point is to split into multiple specialized judge calls rather than one overloaded one. A faithfulness-and-completeness judge focused purely on content correctness, and a separate tone-and-formatting judge focused purely on presentation, each with a tighter, shorter rubric, will usually outperform one mega-judge trying to hold six criteria in its head at once. Yes, this costs more in latency and API spend, but it buys back the reliability that a bloated single-pass rubric loses. Treat "how many dimensions can one judge call handle well" as an empirical question you answer with the same calibration technique from earlier — if judge-vs-human agreement drops as you add dimensions, that's your signal to split.

Closing thoughts

A custom rubric is the difference between an LLM judge that produces defensible, auditable signal and one that produces numbers people quietly stop trusting after the third disagreement. The work is front-loaded: cataloging real failure modes, writing criteria specific enough that two humans would score the same way, and calibrating against actual human judgment before anything gates a release. None of that is exotic — it's the same rigor you'd bring to writing a good test suite, applied to a probabilistic grader instead of a deterministic one.

Once the rubric is solid, LLM-as-a-Judge stops being a hand-wavy shortcut and becomes a real evaluation layer — one you can run on every pull request, every prompt tweak, and every model upgrade, with a documented history of what "good" meant at each point in time.