Building an Eval Team: Roles and Responsibilities

Why "just add evals" is bad advice

Every team shipping an LLM feature eventually hits the same wall. The demo worked. The prompt looked great in the notebook. Then it went to production and started hallucinating customer refund policies, or summarizing tickets in a tone that made support agents cringe, or quietly regressing every time someone tweaked the system prompt. Someone says "we need evals," a junior engineer writes a script that checks for exact string matches, and three weeks later nobody trusts the eval either.

The problem isn't that evals are hard to write. It's that eval work gets treated as a task instead of a function. A task gets assigned to whoever is free on a Tuesday. A function needs owners, handoffs, and a shared understanding of what "good" means for your product. Teams that get LLM quality right — the ones whose models keep improving release after release instead of oscillating — treat evaluation as a discipline with distinct roles, not a checkbox before deploy.

This article is about those roles. Not job titles you need to post on LinkedIn tomorrow, but responsibilities that need a clear owner even if one person wears three hats. If you're building or scaling an AI product, understanding this division of labor will save you from the two most common failure modes: eval theater (metrics that look rigorous but measure nothing real) and eval paralysis (so much process that nobody ships).

The core problem an eval team solves

Before assigning roles, it helps to be precise about what an eval team actually exists to do. It is not to produce a dashboard. It exists to answer one question repeatedly, quickly, and with enough confidence that the team acts on the answer: is this change to the system making outputs better or worse, and for whom?

That question splits into sub-problems that map naturally onto different skill sets:

• Defining what "better" means for this specific product (not generic benchmarks)
• Collecting representative examples of real usage, including the ugly edge cases
• Writing scoring logic that's actually correlated with human judgment
• Running evaluations at a cadence that keeps up with shipping velocity
• Turning eval failures into actionable bug reports engineers can fix
• Watching for drift once the system is live, not just before launch

Trying to have one generalist "AI engineer" own all six of these is how eval efforts collapse. Each sub-problem rewards a different kind of thinking — product judgment, domain expertise, statistics, and software engineering rarely live comfortably in the same brain under deadline pressure.

There's also a sequencing problem hiding in that list. Most teams start writing eval code before they've agreed on what quality means, which is backwards. It's the equivalent of writing unit tests before deciding what the function is supposed to do. The rubric has to exist first, even in rough form, or the scoring logic that gets built on top of it will be measuring whatever was easy to measure rather than what actually matters to users. Keep that ordering in mind as you read through the roles below — they're numbered roughly in the order a new eval effort should staff them, not by seniority or headcount.

Role 1: The Eval Product Owner

This person answers the question nobody else wants to touch: what does quality actually mean here? For a coding assistant, is a "good" answer one that compiles, one that matches idiomatic style, or one that a senior engineer wouldn't need to revise? For a customer support bot, is tone more important than resolution rate?

The Eval Product Owner is usually the PM or a senior domain expert, not an ML engineer. Their job is to translate fuzzy business goals into a rubric that's specific enough to be graded consistently. Concretely, this means writing and maintaining an evaluation rubric document — the thing every other role points back to when there's a disagreement about a score.

A rubric for a support-ticket summarizer might look like this in practice:

Rubric: Ticket Summary Quality (score 1-5 per dimension)

1. Factual grounding
   5 = every claim traceable to the ticket thread
   3 = mostly grounded, one minor unsupported inference
   1 = contains claims not present anywhere in the thread

2. Completeness
   5 = captures issue, attempted fixes, and current status
   3 = captures issue but omits attempted fixes or status
   1 = misses the core issue entirely

3. Actionability
   5 = a new agent could pick up the ticket with zero re-reading
   1 = a new agent would have to re-read the full thread anyway

Without a document like this, every grader — human or LLM — invents their own definition of quality, and your eval scores become noise dressed up as signal. The Eval Product Owner's real output isn't code. It's a shared, testable definition of "good" that survives contact with disagreement.

The hardest part of this role isn't writing the first draft of the rubric — it's revising it under pressure from real disagreements. When two people rate the same output a 5 and a 2, the instinct is to average and move on. Resist that. The disagreement is information: it usually means the rubric dimension is ambiguous, or it's actually measuring two different things bundled into one score. A rubric dimension like "helpfulness" almost always needs to be split into narrower dimensions — factual grounding, completeness, actionability — precisely because raters disagree wildly on the vague version and converge on the specific ones. The Eval Product Owner should expect to revise the rubric two or three times in the first month of real use, and treat that churn as the system working, not as failure to get it right the first time.

It also helps to explicitly rank the rubric dimensions rather than treating them as equally weighted. For a medical-adjacent product, factual grounding might be non-negotiable — a single hallucination fails the whole response regardless of how well-formatted it is. For a creative writing assistant, tone and originality might outweigh strict factual grounding. Writing that priority order down, even informally, prevents the Scoring Specialist from having to guess later when they build an aggregate score out of per-dimension numbers.

Role 2: The Data Curator

Evals are only as good as the examples they run on. The Data Curator's job is building and maintaining the eval set: a representative, versioned collection of inputs (and, where possible, reference outputs or acceptable-answer ranges) that reflects what the system actually encounters in production, not what the team imagined it would encounter.

This role does three things well when it's staffed properly:

• Sources real traffic, not synthetic edge cases invented in a meeting room. Production logs, support tickets, sales call transcripts — the messy stuff.
• Stratifies the set so easy and hard cases are both represented in known proportions, so an eval score doesn't get inflated by testing mostly softball inputs.
• Tags failure categories so when something breaks, you know if it's a retrieval problem, a formatting problem, or a genuine reasoning failure.

A common mistake teams make here is letting the eval set go stale. It gets built once for launch and never touched again, while the product and its user base evolve underneath it. The Data Curator's job includes a recurring cadence — weekly or per-release — of pulling new production samples, especially ones flagged by users or support as bad, into the eval set.

# Simplified sketch of a curated eval-set entry
eval_case = {
    "id": "ticket_summarizer_0417",
    "input": ticket_thread_text,
    "category": "multi_issue_ticket",   # tag for slicing results later
    "difficulty": "hard",
    "source": "production_flagged",     # vs. "synthetic" or "hand_written"
    "reference_notes": "Must mention refund AND shipping delay, both raised by customer",
}

That category and source metadata is what lets you later ask "are we regressing specifically on multi-issue tickets?" instead of staring at one aggregate pass rate that hides the interesting variance.

There's a sizing question every Data Curator eventually faces: how many examples is enough? The honest answer is that it depends on how many categories you're slicing by, not on some universal number like "100 examples." If your eval set has ten categories and you want a stable read on each one, you need enough examples per category — not just in total — to avoid a single weird example swinging that category's score by twenty points. A common mistake is building a 200-example eval set that's actually 15 categories with roughly 13 examples each, then being surprised when the per-category scores bounce around wildly from run to run. It's better to have fewer categories with real statistical weight behind each than a taxonomy so fine-grained it can't support any of them.

The other discipline this role owns is separating the eval set from anything the model might have been trained or fine-tuned on. If your eval examples leak into a fine-tuning set, or if a prompt-engineering iteration was tuned by literally looking at eval set failures and patching around them one by one, the eval stops measuring generalization and starts measuring memorization. The Data Curator should hold back a portion of examples — sometimes called a holdout set — that engineers don't get to see in detail, only aggregate scores on. It's the same discipline as a train/test split, applied to prompts and rubrics instead of model weights.

Role 3: The Eval Engineer

This is the person who turns the rubric and the data set into something that runs automatically. The Eval Engineer builds the harness: the code that feeds inputs to the model, captures outputs, applies scoring logic, and produces a report. They own correctness of the pipeline itself — making sure the eval isn't silently broken, timing out, or scoring the wrong field.

A minimal harness looks something like this:

import json

def run_eval(model_fn, eval_cases, scorer_fn):
    results = []
    for case in eval_cases:
        output = model_fn(case["input"])
        score = scorer_fn(case, output)
        results.append({
            "id": case["id"],
            "category": case["category"],
            "score": score,
            "output": output,
        })
    return results

def summarize(results):
    by_category = {}
    for r in results:
        by_category.setdefault(r["category"], []).append(r["score"])
    return {
        cat: sum(scores) / len(scores)
        for cat, scores in by_category.items()
    }

if __name__ == "__main__":
    with open("eval_cases.json") as f:
        cases = json.load(f)
    results = run_eval(my_model, cases, my_scorer)
    report = summarize(results)
    print(json.dumps(report, indent=2))

Nothing exotic — the value isn't in clever code, it's in reliability. The Eval Engineer makes sure this pipeline runs on every pull request or nightly, that it's fast enough people actually wait for it, and that a failing eval produces a diffable output (which examples regressed, not just a single aggregate number). If engineers have to manually run a notebook to check eval scores, the eval effort is already dying — automation is this role's whole reason to exist.

Two engineering details separate a harness people actually use from one that gets ignored. First, speed: if a full eval run takes two hours, engineers will stop running it before merging and start running it "later," which in practice means never. Parallelizing model calls, caching outputs for unchanged inputs, and running a fast subset on every commit with the full set nightly are all reasonable trade-offs to keep the feedback loop tight. Second, diffability: a report that says "score dropped from 0.84 to 0.79" is nearly useless on its own. A report that says "these 11 specific example IDs flipped from pass to fail, here's the before/after output for each" is what actually gets acted on. The Eval Engineer's job is making the second kind of report the default, not an afterthought someone has to dig for.

It's also worth this role owning basic reproducibility hygiene: pinning model versions and temperature settings used during eval runs, storing raw outputs alongside scores (not just the scores), and versioning the eval harness itself so that a score from three months ago can be meaningfully compared to a score today. Without that discipline, "the eval score improved" can just as easily mean "the model got better" or "the harness silently changed how it truncates long inputs" — and nobody will be able to tell which.

Role 4: The Scoring/Judge Specialist

Scoring model outputs is its own specialty, distinct from building the harness that runs them. Some things can be scored with plain code — did the JSON parse, is the phone number in the right format, is the response under the token limit. Those are cheap and deterministic and should always be preferred when possible.

But most interesting quality questions — is this response helpful, is this summary faithful, is this tone appropriate — can't be captured by a regex. That's where the Scoring Specialist designs and validates LLM-as-a-Judge setups: using a separate (often stronger) model to grade outputs against the rubric the Eval Product Owner defined.

This role's core discipline is validating the judge itself before trusting it. A judge prompt that hasn't been checked against human ratings is just automated guessing with extra steps. A reasonable validation loop:

def judge_prompt(rubric, input_text, model_output):
    return f"""
You are grading a response against this rubric:
{rubric}

Input:
{input_text}

Response to grade:
{model_output}

Return a JSON object: {{"score": <1-5>, "reasoning": "<one sentence>"}}
Be strict. Do not give a 5 unless every rubric criterion is fully met.
"""

def validate_judge_agreement(judge_fn, human_labeled_set):
    agreements = 0
    for case in human_labeled_set:
        judge_score = judge_fn(case["input"], case["output"])
        if abs(judge_score - case["human_score"]) <= 1:
            agreements += 1
    return agreements / len(human_labeled_set)

If that agreement rate against a small human-labeled sample is low, the fix is not to give up on LLM-as-a-Judge — it's to iterate on the judge prompt, tighten the rubric language, or switch to a stronger judge model, then re-validate. Teams that skip this validation step end up with a judge that's confidently wrong, which is worse than no automated scoring at all because it creates false confidence.

A few specific failure modes come up often enough that this role should watch for them by name. Position bias shows up in pairwise comparisons — when a judge is asked to pick the better of two responses, it will systematically favor whichever one is shown first or second, regardless of content, unless you randomize order and average across both orderings. Length bias is the tendency of judges to rate longer responses as more thorough even when the extra length is padding rather than substance; a good validation set should include a few cases where the shorter answer is objectively better, specifically to catch this. Self-preference bias shows up when the judge model and the model being evaluated are the same family — the judge tends to rate outputs in its own house style more favorably. Using a different model (or at minimum a different prompt lineage) as the judge is a cheap mitigation.

The Scoring Specialist should also decide, deliberately, whether a single blended score or a vector of per-dimension scores is more useful to report. Blended scores are easier to put on a dashboard and easier to set a single pass/fail threshold on. Per-dimension scores are more diagnostic — they tell you whether a regression is about factual grounding or about tone, which matters enormously for triage. In practice, most mature eval setups report both: a headline blended number for quick health checks, and the per-dimension breakdown for anyone who needs to actually fix something.

Role 5: The Triage and Bug-Reporting Owner

An eval run that produces a score with no path to action is a wasted eval run. Someone has to own turning "category X dropped from 92% to 81%" into a specific, reproducible bug that an engineer can pick up and fix. This is triage — and it's a distinct skill from writing the eval in the first place.

Good triage means reading the actual failing transcripts, not just the aggregate number, and clustering failures by root cause. Ten failures might collapse into two actual bugs: a prompt template that drops context when the ticket has more than one attachment, and a retrieval step that returns stale documents for a specific product line. Without this role, engineers get handed a spreadsheet of scores and have to redo the triage work themselves, usually badly, usually under time pressure, usually skipping it.

A useful practice here is a lightweight failure taxonomy that gets attached to every low-scoring case:

Failure categories:
- HALLUCINATION: states something not supported by input
- OMISSION: misses required information present in input
- FORMAT: correct content, wrong structure/format
- TONE: content and structure fine, register is wrong
- LATENCY: correct output, exceeded time/token budget

Tagging failures this way turns a vague "quality dropped" signal into "12 new HALLUCINATION failures, all involving multi-attachment tickets" — which is something a specific engineer can act on in an afternoon instead of a debugging spiral.

Role 6: The Production Monitoring Owner

Everything above happens mostly pre-deploy, against a fixed eval set. But production drifts. User behavior shifts, upstream models get silently updated by vendors, new categories of requests show up that were never in your eval set. Someone needs to own catching this after launch, not just before it.

This role's job is sampling live traffic, running lightweight automated scoring (often the same LLM-as-a-Judge setup from Role 4, run continuously rather than on a fixed set) on a rolling basis, and alerting when scores move outside expected bounds. It's also the role that feeds new hard cases back to the Data Curator — closing the loop so the eval set doesn't go stale.

def monitor_production_sample(sample_batch, judge_fn, baseline_mean, threshold=0.15):
    scores = [judge_fn(x["input"], x["output"]) for x in sample_batch]
    current_mean = sum(scores) / len(scores)
    drop = baseline_mean - current_mean
    if drop > threshold:
        alert(f"Quality drop detected: {drop:.2f} below baseline")
        flag_for_eval_set(sample_batch)  # send hardest cases back to curation
    return current_mean

Without an owner for this role, quality regressions get discovered by angry users or a churn spike weeks after the actual cause — usually a vendor model update or a config change nobody connected to the drop in quality.

How these roles work together in practice

On a small team, one or two people cover several of these roles, and that's fine — the point isn't headcount, it's making sure each responsibility has an explicit owner and doesn't fall through the cracks between "the PM assumed engineering had it" and "engineering assumed the PM had it." A realistic small-team mapping looks like this: the PM owns the rubric and triage, one engineer owns the harness and the judge validation, and the whole team rotates through reviewing production samples weekly.

The failure pattern to watch for is when the same person owns both the rubric definition and the scoring implementation with no check between them — they'll unconsciously write a judge prompt that confirms their own intuitions rather than testing them. Even on a two-person team, it's worth having someone else spot-check the judge against a handful of cases before trusting it.

Staffing this incrementally instead of all at once

Nobody builds all six roles in week one, and trying to is its own mistake — you'll spend a month writing process documents before a single real eval runs. A more realistic path looks like three stages.

Stage one, pre-launch: one person, usually the founding engineer or PM, does a rough version of the Eval Product Owner and Data Curator roles together. The rubric is a paragraph, not a document. The eval set is thirty hand-picked examples, not three hundred stratified ones. The goal here is just to stop shipping changes based on vibes — a crude eval beats no eval.

Stage two, early production: as real usage starts, the Eval Engineer role becomes necessary because manually re-running thirty examples in a notebook every time someone changes a prompt stops scaling within weeks. This is also when the Scoring Specialist earns its keep, because eyeballing outputs by hand turns into hundreds of outputs nobody has time to read.

Stage three, scaling production: Triage and Production Monitoring become explicit responsibilities rather than something the on-call engineer does in a panic after a user complaint. By this stage the eval set has grown into hundreds or low thousands of examples, the judge has been through several rounds of validation, and the team knows what normal week-over-week fluctuation looks like versus a real regression.

The mistake to avoid at every stage is skipping straight to stage three's tooling — dashboards, multi-judge ensembles, automated alerting — before stage one's basic question has been answered: does anyone agree on what a good output looks like for this product? Sophisticated infrastructure around an undefined notion of quality just produces precise-looking numbers that mean nothing.

Closing thoughts

None of these roles are exotic. They're closer to a QA discipline than a research discipline — curation, harness-building, scoring validation, triage, and monitoring are all familiar shapes if you've shipped software before. What's new is that one of the "testers" in this pipeline is itself a language model, which is exactly why the Scoring/Judge Specialist role matters as much as it does. LLM-as-a-Judge is now the backbone of most production eval pipelines because writing exhaustive rule-based checks for open-ended text doesn't scale — but a judge that hasn't been validated against real human ratings is just a second unverified model grading a first one. Get the roles right, validate the judge, and the rest of the eval pipeline stops being guesswork and starts being an actual engineering practice.