Building a Golden Dataset for LLM Evaluation

Why your evals are lying to you

You shipped a prompt change. The eval score went up. You deployed. Users started complaining within a week. Sound familiar? This is the most common failure mode in LLM engineering teams right now, and it almost never traces back to the model. It traces back to the dataset you used to measure the model.

Most teams treat evaluation data as an afterthought — a folder of twenty examples someone wrote in an afternoon, mixed with a handful of support tickets pasted in before a demo. Then they run GPT-4 or Claude as a judge over that folder and treat the resulting score as ground truth. The problem is that a bad dataset produces a confident, precise-looking number that means nothing. You can optimize a prompt for weeks against twenty unrepresentative examples and get a beautiful eval curve while your production quality quietly degrades.

A golden dataset is the fix. It's a curated, versioned, human-verified set of inputs and expected outputs that represents the real distribution of what your system needs to handle — including the edge cases that break it. Building one properly is unglamorous work: no clever prompt engineering, no model architecture decisions, just careful labeling and constant maintenance. But it is the single highest-leverage investment you can make in an LLM product, because every other evaluation technique — LLM-as-a-judge, automated regression testing, A/B test gating — is only as good as the data it runs against.

This article walks through how to actually build one: sourcing examples, defining what "good" means, writing the labeling process, structuring the data, and keeping it alive as your product evolves.

What a golden dataset actually is

A golden dataset is not just "some example inputs." It has four properties that separate it from an ad hoc test folder.

It is representative. The distribution of inputs in your dataset should mirror the distribution your system sees in production — not the distribution that's easiest to collect or the distribution that makes your model look good. If 30% of your real user queries are ambiguous or underspecified, 30% of your golden set should be too.

It has a defined ground truth. Every example needs an expected output, or at minimum a rubric describing what a correct output looks like. For classification or extraction tasks this might be an exact expected value. For open-ended generation, it's usually a set of criteria plus one or more reference answers.

It is versioned. Datasets change as your product changes. You need to know which version of the dataset produced which eval score, the same way you version code.

It is adversarially maintained. New failure modes get added as you find them in production. A golden dataset that never grows is a golden dataset that's slowly going stale.

Here's a minimal schema that captures these properties for a single example:

{
  "id": "gd-0142",
  "input": {
    "query": "Can I get a refund if I cancel my subscription after the trial ends?",
    "context": {
      "user_plan": "pro_monthly",
      "trial_status": "ended_3_days_ago"
    }
  },
  "expected_output": {
    "answer_contains": [
      "no refund for the current billing period",
      "cancellation takes effect at period end"
    ],
    "must_not_contain": [
      "full refund",
      "immediate refund"
    ],
    "category": "billing_policy",
    "difficulty": "medium"
  },
  "source": "production_ticket",
  "date_added": "2026-03-14",
  "tags": ["refunds", "trial", "billing"],
  "reviewed_by": "ira.menon"
}

Notice this isn't just a string match. The expected_output is a rubric — a set of things that must appear and must not appear — because for generative tasks, exact string equality is almost always the wrong bar.

Sourcing examples: where good data actually comes from

The single biggest mistake teams make is writing all their eval examples from imagination. Engineers are bad at predicting what users will actually type. You need at least three sources, and ideally all four.

1. Production logs. This is your best source once you have any real traffic. Sample real queries — including the weird, malformed, and multi-intent ones — and turn a subset into golden examples. If you have no production traffic yet, use logs from a closed beta, a support inbox, or even a competitor's public forum where users describe similar tasks.

2. Subject matter expert authored examples. For domains with real correctness stakes — legal, medical, financial, technical support — have a domain expert write examples deliberately targeting known tricky cases. An SME knows where the traps are; an engineer usually doesn't.

3. Adversarial / red-team examples. Deliberately construct inputs designed to break your system: prompt injection attempts, contradictory instructions, extremely long context, missing information, multilingual inputs if you claim to support them, and boundary cases in your business logic (what happens exactly at the trial cutoff?).

4. Regression cases from real incidents. Every time production breaks — a hallucinated policy, a wrong calculation, an unsafe response — that failing input becomes a permanent golden example. This is how the dataset earns its keep over time; it turns every incident into a permanent tripwire.

A rough allocation that works well for a customer-support-style LLM product: 50% production-sampled, 20% SME-authored edge cases, 15% adversarial, 15% incident-derived. Adjust based on how mature your product is — a pre-launch product will lean much more heavily on SME and adversarial examples simply because production logs don't exist yet.

Defining ground truth without pretending everything is deterministic

Most teams get stuck here because they try to force every task into an exact-match format. Don't. Match your ground-truth format to the task type.

Exact match — for classification, extraction of structured fields, routing decisions. "This ticket should be tagged billing" is either right or wrong.

Set-based match — for extraction where order doesn't matter or multiple valid phrasings exist. "The response must mention these three data points" rather than a full-string comparison.

Rubric-based match — for open-ended generation. Instead of one canonical answer, define a checklist: Does it address the user's actual question? Does it avoid making unsupported claims? Does it match brand tone? Does it correctly cite the source policy? This is where LLM-as-a-Judge comes in later, but the rubric itself is authored by a human and stored as part of the golden example, not invented by the judge model at eval time.

Reference answer with semantic similarity — for tasks like summarization where there's no single correct output but a family of acceptable ones. You store one or more reference summaries and compare against them using either embedding similarity or a judge model scoring against the reference.

Here's how you might structure a rubric-based example for a RAG-based support bot:

golden_example = {
    "id": "gd-0201",
    "input": {
        "query": "Why was I charged twice this month?",
        "retrieved_docs": ["billing_faq_v3", "proration_policy_v2"]
    },
    "rubric": [
        {
            "criterion": "acknowledges_the_specific_issue",
            "description": "Response directly addresses the double-charge concern, not a generic billing reply",
            "weight": 3
        },
        {
            "criterion": "explains_proration_if_applicable",
            "description": "If a plan change occurred, response explains proration as likely cause",
            "weight": 2
        },
        {
            "criterion": "no_fabricated_refund_promise",
            "description": "Does not promise a refund unless policy explicitly allows one",
            "weight": 5
        },
        {
            "criterion": "provides_next_step",
            "description": "Gives a clear action: link to billing history, or escalation path",
            "weight": 2
        }
    ],
    "max_score": 12,
    "pass_threshold": 9
}

The weight on no_fabricated_refund_promise is deliberately high. This is the pattern experienced teams converge on: not all criteria are equal, and safety-critical or trust-critical criteria should dominate the score so a model can't pass by being fluent while getting the one thing that matters wrong.

Labeling process: getting humans to agree on ground truth

A rubric is only useful if humans can apply it consistently. If two reviewers score the same output differently, your eval is noise regardless of how good the dataset looks on paper.

Run an inter-rater reliability check before you trust any rubric at scale. Take 30-50 examples, have two people independently score them against the rubric, and measure agreement. For binary pass/fail criteria, simple percentage agreement is fine as a first pass; for anything you plan to publish or use to gate deployment, use Cohen's kappa so you're accounting for chance agreement.

from sklearn.metrics import cohen_kappa_score

# 1 = criterion met, 0 = criterion not met, per example
reviewer_a = [1, 1, 0, 1, 0, 1, 1, 0, 0, 1]
reviewer_b = [1, 0, 0, 1, 0, 1, 1, 0, 1, 1]

kappa = cohen_kappa_score(reviewer_a, reviewer_b)
print(f"Cohen's kappa: {kappa:.2f}")
# below ~0.6 usually means the rubric or criterion definitions are ambiguous

If kappa comes back low, the fix is almost never "get stricter reviewers" — it's "the criterion is underspecified." Rewrite the criterion with a concrete example of a passing and a failing response, then re-run the check. This iteration loop — write rubric, test agreement, refine wording, retest — is the actual bulk of the labeling effort, and it's worth budgeting real time for it rather than treating it as a formality.

Once agreement is acceptable, a practical workflow for scaling labeling looks like this:

1. One person drafts the initial expected output or rubric for each new example.
2. A second person reviews and either approves or flags disagreement.
3. Disagreements get resolved in a short sync, and the resolution gets written back into the rubric as a clarifying note so the next similar case doesn't cause the same disagreement.
4. Approved examples get a reviewed_by and review_date field before they're allowed into the dataset used for gating deployments.

Never let unreviewed examples into the golden set that gates releases. It's fine to have a staging pool of candidate examples awaiting review — just don't blend it with the trusted set.

Sizing and structuring the dataset

A question that comes up constantly: how many examples do you actually need? There's no universal number, but here's a grounded way to think about it.

For each distinct task category or intent your system handles, you want enough examples to detect a meaningful regression with reasonable confidence — in practice this tends to land somewhere between 20 and 50 examples per category for most product-scale evals, more if the category is high-stakes or high-volume. If you have 15 intent categories, that's 300-750 examples minimum, not counting adversarial and edge cases layered on top.

Structure the dataset with explicit slicing so you can see where quality breaks down, not just an aggregate score:

dataset_slices:
  by_category:
    - billing_policy
    - account_management
    - technical_troubleshooting
    - refund_requests
  by_difficulty:
    - easy       # single clear intent, complete information
    - medium     # some ambiguity or missing context
    - hard       # multi-intent, contradictory, or adversarial
  by_source:
    - production_sampled
    - sme_authored
    - adversarial
    - incident_derived

When you run an eval, report scores per slice, not just an overall average. An aggregate score of 91% can hide the fact that your hard slice is at 60% and your refund_requests category — the one with real financial consequences — is failing at twice the rate of everything else. This is usually the single most actionable output of a well-structured golden dataset: it tells you exactly where to focus the next round of prompt or retrieval fixes, instead of leaving you staring at one number wondering what to do next.

Versioning and change management

Treat the golden dataset like code. Store it in version control, not in a spreadsheet someone edits by hand and forgets to notify the team about.

golden-datasets/
├── v1.0.0/
│   ├── examples.jsonl
│   ├── rubrics.yaml
│   └── CHANGELOG.md
├── v1.1.0/
│   ├── examples.jsonl
│   ├── rubrics.yaml
│   └── CHANGELOG.md
└── current -> v1.1.0

Every eval run should record which dataset version it used, alongside the model version, prompt version, and any retrieval index version. Without this, you cannot answer the most important question in LLM engineering: "did the score change because the model got better, or because the test changed?"

{
  "eval_run_id": "run-2026-07-01-0842",
  "model": "claude-sonnet-4.5",
  "prompt_version": "v12",
  "dataset_version": "v1.1.0",
  "retrieval_index_version": "2026-06-28",
  "aggregate_score": 0.89,
  "slice_scores": {
    "billing_policy": 0.94,
    "refund_requests": 0.81,
    "hard_difficulty": 0.72
  }
}

When you bump the dataset version, write a changelog entry explaining what changed and why — new examples added from an incident, a rubric criterion reworded after a kappa check failed, an outdated example removed because the underlying product behavior changed. This changelog is what lets you distinguish real model regressions from dataset drift when a score suddenly moves.

Keeping the dataset alive instead of letting it rot

A golden dataset that was perfect on day one and untouched since is now measuring a product that no longer exists. Build maintenance into your regular workflow, not as a special project.

Add a golden example every time production breaks. This is the highest-value habit on this list. The moment you find a real failure — a hallucination, a wrong policy statement, an unsafe response — write it into the dataset with the corrected expected output before you fix the underlying issue. Now you have a permanent regression guard for that exact failure mode.

Audit for staleness quarterly. Product policies change, pricing changes, features get deprecated. An example whose expected output references a policy that no longer exists isn't neutral — it actively penalizes a model for being correct. Schedule a recurring review, even a lightweight one, where someone checks a sample of the dataset against current product truth.

Watch for reviewer drift. If the same two people have been labeling for six months, re-run the inter-rater reliability check periodically. People's calibration shifts over time, often without anyone noticing.

Retire examples that stop discriminating. If every model version you've tested for the last year scores 100% on an example, it's not telling you anything anymore. Keep it in an archive for documentation purposes, but move it out of the active gating set so it isn't diluting your signal.

Guard against overfitting to the dataset itself. If your team starts prompt-engineering specifically to pass known golden examples rather than to solve the underlying task, the dataset stops being a valid measurement — this is Goodhart's Law showing up in your eval pipeline. The fix is to hold out a portion of the dataset that only a small group can see, and rotate which examples are visible for iterative prompt development versus which are reserved purely for release gating.

Putting it together in a real pipeline

Here's a simplified but complete example of how a golden dataset gets consumed in an actual CI-style eval run, checking exact-match fields and rubric criteria together:

import json

def load_golden_dataset(path):
    with open(path) as f:
        return [json.loads(line) for line in f]

def run_exact_checks(example, model_output):
    expected = example["expected_output"]
    passed = all(
        phrase.lower() in model_output.lower()
        for phrase in expected.get("answer_contains", [])
    )
    forbidden = any(
        phrase.lower() in model_output.lower()
        for phrase in expected.get("must_not_contain", [])
    )
    return passed and not forbidden

def score_example(example, model_output, judge_fn):
    if "rubric" in example:
        # delegate open-ended scoring to an LLM judge
        return judge_fn(example, model_output)
    return 1.0 if run_exact_checks(example, model_output) else 0.0

def run_eval(dataset_path, generate_fn, judge_fn):
    dataset = load_golden_dataset(dataset_path)
    results = []
    for example in dataset:
        output = generate_fn(example["input"])
        score = score_example(example, output, judge_fn)
        results.append({"id": example["id"], "score": score})
    return results

The judge_fn here is where LLM-as-a-Judge enters the picture — but notice it only shows up at the very end of this whole process, scoring against a rubric that humans already wrote, reviewed, and agreed on. That ordering matters. An LLM judge evaluated against a sloppy, unrepresentative, unreviewed dataset just produces a confident-sounding wrong answer faster. Get the golden dataset right first — sourced from real usage, carefully labeled, versioned, and continuously fed by production incidents — and the judge model becomes a genuinely reliable amplifier of human judgment instead of a fig leaf over an eval process nobody trusts. That's the actual payoff: not a fancier scoring mechanism, but an eval you can act on with confidence.