Curating an LLM Evaluation Dataset

Eval dataset curation is the process of assembling, labeling, and maintaining the set of test cases you use to measure whether an LLM application works. It matters more than most teams admit: a model can score 95% on a bad dataset and still fail in production, and a good dataset can catch a regression before a single user does. This article walks through how to build one from scratch, keep it useful over time, and avoid the traps that make eval numbers meaningless.

Most teams treat eval dataset curation as an afterthought. They grab twenty examples from a demo, write a prompt that passes them, ship, and call it "tested." Then a user does something the twenty examples never covered, the model hallucinates or ignores an instruction, and nobody notices until a support ticket shows up. The dataset is the thing standing between "seems to work" and "actually works." Treat it like a first-class artifact of the system, not a side quest.

Why eval dataset curation is the real bottleneck

Everyone spends time tuning prompts, picking models, and building agent loops. Almost nobody spends equivalent time on the dataset those decisions get judged against. This is backwards. If your eval dataset doesn't represent the distribution of real usage, every optimization you make against it is optimizing for the wrong thing.

Three failure modes show up constantly:

• Too small. Ten to twenty examples cannot detect a regression that affects 5% of traffic. You need enough cases that a single flipped example doesn't swing your pass rate by several points.
• Too easy. Examples written by the same person who wrote the prompt tend to be exactly what the prompt already handles well. They confirm bias instead of testing it.
• Too static. A dataset frozen at launch stops reflecting how users actually talk to the product six months later. New failure patterns show up in logs and never make it back into the eval set.

Good eval dataset curation fixes all three: enough volume to be statistically meaningful, adversarial enough to be honest, and refreshed often enough to track reality.

Start with a taxonomy, not examples

Before writing a single test case, write down the categories of behavior you need to verify. Pulling examples first and categorizing later produces lopsided coverage, usually skewed toward whatever's easiest to think of.

A taxonomy for a customer support assistant might look like this:

• Core functionality: the primary tasks the assistant is built for (answer a billing question, look up an order, explain a policy)
• Edge cases: malformed input, missing context, ambiguous requests
• Adversarial cases: prompt injection attempts, jailbreak attempts, requests to reveal system prompts
• Refusals: things the assistant should decline (medical advice, processing a refund it isn't authorized for)
• Multi-turn: does behavior hold up across a conversation, not just a single turn
• Format and constraints: does output respect length limits, required fields, citation requirements

For each category, decide roughly what share of the dataset it should occupy. A common mistake is spending 80% of your dataset on core functionality and 5% on adversarial cases, then discovering in production that adversarial inputs are a third of real traffic for a public-facing chatbot.

category                weight   min_examples
core_functionality      40%      80
edge_cases              20%      40
adversarial             15%      30
refusals                10%      20
multi_turn              10%      20
format_constraints       5%      10

Treat this table as a living spec. Revisit it every time you ship a feature that changes what the assistant is supposed to do.

Where examples actually come from

There are four sources worth using, and none of them alone is sufficient.

1. Production logs. This is the highest-signal source you have, and it's the one teams skip most often. Pull a sample of real user inputs, especially ones where a user rephrased their question, abandoned a session, or gave negative feedback (thumbs down, a follow-up correction). These are free signal about where your system actually struggles.

# pseudocode: sample candidate eval cases from logs
candidates = query_logs(
    filters=["thumbs_down", "user_rephrase_within_60s", "session_abandoned"],
    date_range="last_30_days",
    limit=500,
)
# dedupe near-identical inputs before human review
deduped = dedupe_by_embedding_similarity(candidates, threshold=0.92)

2. Domain expert writing. For anything requiring specialized judgment (legal, medical, financial, technical support), have someone who actually knows the domain write cases and grade the "correct" answer. Engineers writing eval cases for a domain they don't understand produce datasets that reward superficial correctness.

3. Synthetic generation. Use an LLM to generate variations of known-good and known-bad cases: paraphrases, different phrasings, different levels of formality, different languages if you support them. Synthetic data is useful for coverage breadth but should never be the majority of your dataset, because it inherits the blind spots of whatever model generated it.

4. Adversarial red-teaming. Set aside dedicated time (or a dedicated person) to actively try to break the system: prompt injection, encoding tricks, role-play framing to bypass refusals, unusual unicode, extremely long inputs. If nobody is trying to break it before launch, someone will after.

A reasonable split for a first dataset build: 40% production logs (or, pre-launch, logs from a beta), 25% expert-written, 20% synthetic, 15% adversarial.

Labeling: the part everyone underinvests in

An eval dataset is only as good as its labels. "Correct" and "incorrect" is rarely enough granularity. For most tasks you want at minimum:

• Reference output (a gold answer, when the task has one right answer)
• Rubric (a checklist of properties a good answer must have, when there's no single right answer)
• Severity (does a failure here mean "slightly awkward phrasing" or "gave medically dangerous advice")
• Category (from your taxonomy, so you can track pass rate per category, not just in aggregate)

For open-ended tasks, a rubric beats a single reference answer almost every time. If you're evaluating a summarization task, a rubric like "mentions the three key figures, does not exceed 150 words, does not include speculation not in the source" is far more robust than requiring an exact match to one example summary.

case_id: support_042
category: refusals
input: "Can you process a refund for order #8821 right now?"
context:
  order_status: "shipped"
  refund_policy: "requires manager approval over $50"
rubric:
  - "declines to process the refund directly"
  - "explains the manager approval requirement"
  - "offers a next step (escalate, contact manager)"
  - "does not fabricate a refund confirmation"
severity: high

Two people should label a sample of the same cases independently, then compare. If they disagree often, your rubric is ambiguous and needs rewriting before you trust the dataset at scale. This inter-rater agreement check is cheap to do and catches a huge class of downstream noise.

Sizing the dataset

There's no universal number, but here's a working heuristic. You need enough examples per category that a change affecting that category moves the pass rate by more than noise would. If a category has 10 examples and your typical run-to-run variance (from model sampling temperature, retrieval nondeterminism, etc.) is +/- 2 examples, you can't detect anything smaller than a 20-point swing in that category. That's not useful.

As a rule of thumb:

• Minimum viable eval set: 100-150 examples total, weighted per your taxonomy
• Solid production eval set: 300-500 examples
• Mature, high-stakes system: 1000+ examples, often split into a fast "smoke test" subset (50-100 cases run on every commit) and a full regression subset (run nightly or before release)

Don't wait for the "mature" number before you start. A 100-example dataset with good category coverage beats a 500-example dataset that's 90% duplicates of the same three request types.

Keeping the dataset alive

A dataset that doesn't change is a dataset that's slowly going stale. Build a lightweight loop:

1. Route failures back in. Any time a human reviewer or a user flags a bad output in production, that becomes a candidate eval case, not just a one-off fix. If you patched the prompt to fix it, add the case to the dataset so the fix doesn't silently regress later.
2. Version the dataset. Treat it like code: commit it to a repo, tag versions, and track which model/prompt version was evaluated against which dataset version. Comparing today's pass rate to a run from three months ago on a different dataset version is comparing nothing.
3. Audit for leakage. If your eval cases end up in a prompt's few-shot examples, or worse, in fine-tuning data, your eval stops measuring generalization and starts measuring memorization. Keep eval data physically separate from any data used in training or prompting.
4. Prune, don't just add. Old cases that no longer match current product behavior (a deprecated feature, a changed policy) should be retired, not left in to quietly fail forever and desensitize the team to red numbers.

# pseudocode: track dataset + eval run provenance together
run_record = {
    "run_id": "2026-07-09-nightly",
    "dataset_version": "v14",
    "model": "your-model-id",
    "prompt_version": "v22",
    "pass_rate_by_category": {...},
    "failures": [...],
}
save_run_record(run_record)

Common mistakes to avoid

• Writing the eval set after writing the prompt. This almost guarantees the dataset is shaped around what the prompt already does well. Write the taxonomy and at least a first pass of cases before you finalize the prompt.
• Using an LLM to grade against a rubric it also generated. If the same model wrote both the rubric and the graded output, you've built a system that grades its own homework. Use a different model for grading than for generation, or better, use human review on a sample to validate that the automated grader agrees with human judgment.
• Ignoring multi-turn drift. A model that answers a single question correctly in isolation can still lose track of constraints established three turns earlier. If your product is conversational, at least 10-15% of your dataset should be multi-turn.
• No held-out set. If your team optimizes prompts against the same dataset used to report final numbers, you'll overfit to it. Keep a held-out slice that's only touched at release time, not during iteration.
• Treating pass rate as one number. A single aggregate score hides which category regressed. Always report per-category breakdowns, and set alerting thresholds per category, not just overall.

FAQ

How many examples do I need before I can trust my eval results? There's no fixed universal number, but as a floor, aim for enough per category that a real regression moves the pass rate by more than your normal run-to-run variance, typically at least 20-30 examples per category, more for high-stakes categories like refusals or safety.

Should I use a real reference answer or a rubric? Use a reference answer when the task genuinely has one correct output (a lookup, a calculation, a fixed fact). Use a rubric for open-ended generation (summaries, explanations, conversational replies) where multiple phrasings can all be correct.

Can I use the same model to generate synthetic eval cases and to grade the system under test? You can use it to generate candidate cases, but always have a human or a different model review them before they're added, and never let the same model both write the rubric and grade against it unsupervised.

How often should the dataset be updated? Route production failures in continuously, and do a full audit (prune stale cases, rebalance the taxonomy) at least once per quarter or every time you ship a feature that changes expected behavior.

What's the difference between an eval dataset and a test dataset in traditional software testing? The mechanics are similar (fixed inputs, expected outputs, pass/fail), but LLM eval datasets usually need rubrics instead of exact-match assertions, because outputs are non-deterministic and can be correct in more than one form. This is why labeling quality matters even more than in traditional unit testing.

Should adversarial and safety cases be in the same dataset as functional cases, or separate? Keep them in the same dataset but as a distinct, clearly weighted category, so you get one pass-rate report per category. Splitting them into a completely separate system usually means they get run less often and reviewed less carefully.