LLM Evaluationgolden datasetLLM testingprompt engineeringevals

Building a Golden Dataset for LLM Evaluation

Pramod Dutta · Jun 30, 2026 · 10 min read

Every serious LLM evaluation setup starts with one thing: an llm golden dataset. Without it you are eyeballing outputs, arguing in Slack about whether a response "feels right," and shipping prompt changes on vibes. A golden dataset is a curated, versioned set of inputs paired with expected outputs (or scoring criteria) that you run your model against every time something changes: a prompt edit, a model swap, a new retrieval pipeline, a temperature tweak. This article walks through building one from zero, with runnable code, no filler.

What a golden dataset actually is

A golden dataset is not a giant pile of production logs. It is a small, deliberately chosen, high-signal set of examples that cover:

Happy path cases: the everyday requests your system must nail every time.
Edge cases: ambiguous phrasing, multi-intent requests, adversarial inputs, empty or malformed fields.
Regression cases: bugs you already fixed once. If you fixed it, it goes in the dataset, permanently.
Hard negatives: inputs that look like they should trigger a behavior but should not (this catches over-eager tool calls or hallucinated confidence).

Size matters less than most teams think. Twenty to eighty tightly curated examples per task category beats a thousand scraped and unreviewed conversation logs. A golden dataset that nobody trusts because it is full of noise is worse than no dataset at all, because it gives false confidence.

Step 1: Define the schema before you write a single example

Every example needs a consistent structure so your eval harness can process it programmatically. Here is a schema that works for most text-generation, classification, and tool-calling tasks:

{
  "id": "refund-policy-001",
  "category": "customer_support",
  "input": {
    "user_message": "Can I get a refund if I bought the course 40 days ago?",
    "system_context": "refund_policy: 30 days from purchase"
  },
  "expected": {
    "must_contain": ["30 day", "not eligible"],
    "must_not_contain": ["yes, you can get a refund"],
    "reference_answer": "Unfortunately this purchase is outside our 30-day refund window, so it is not eligible for a refund.",
    "scoring_rubric": "Correctly states the 30-day window and denies the refund without being rude or apologetic to excess."
  },
  "metadata": {
    "difficulty": "medium",
    "added_by": "pramod",
    "added_date": "2026-06-14",
    "source": "support_ticket_4821",
    "tags": ["refund", "policy", "negative_case"]
  }
}

Keep four fields non-negotiable: id (stable, never reused), input, expected, and metadata.tags. The tags are what let you slice results later ("show me every failure tagged refund") without re-annotating anything.

Step 2: Source examples from real usage, not your imagination

The fastest way to build a weak golden dataset is to sit down and invent forty questions you think users might ask. Real users are more creative and more sloppy than you are. Pull from:

Production logs: filter for low user satisfaction signals (thumbs down, regenerate clicks, session abandonment right after a response).
Support tickets: anywhere a human had to step in because the model got it wrong is gold.
Internal dogfooding: your own team's usage, especially the requests that made someone say "wait, that's wrong" out loud.
Adversarial red-teaming: deliberately try to break the system (prompt injection, contradictory instructions, out-of-scope requests).

A simple script to pull candidate examples from a logs table and stage them for review:

import csv
import json
from datetime import datetime, timedelta

def pull_candidates(logs_path, days_back=30, min_length=10):
    cutoff = datetime.now() - timedelta(days=days_back)
    candidates = []

    with open(logs_path, newline="", encoding="utf-8") as f:
        reader = csv.DictReader(f)
        for row in reader:
            ts = datetime.fromisoformat(row["timestamp"])
            if ts < cutoff:
                continue
            if row.get("feedback") not in ("thumbs_down", "regenerated"):
                continue
            if len(row["user_message"]) < min_length:
                continue
            candidates.append({
                "id": f"candidate-{row['session_id']}",
                "input": {"user_message": row["user_message"]},
                "model_output": row["model_response"],
                "feedback": row["feedback"],
                "metadata": {"source": "production_logs", "reviewed": False}
            })

    return candidates

if __name__ == "__main__":
    results = pull_candidates("logs_export.csv")
    with open("candidates.json", "w") as out:
        json.dump(results, out, indent=2)
    print(f"Staged {len(results)} candidates for human review")

Note the "reviewed": False flag. Nothing from this script goes into the actual golden dataset until a human writes the expected block by hand. Automated harvesting gets you candidates, not ground truth.

Step 3: Write expected outputs with rubrics, not just exact strings

Exact-match expected outputs only work for narrow tasks like classification or extraction. For open-ended generation, use a rubric-based expected block that an LLM judge (or a human) can score against. This is the single biggest lever for making a golden dataset useful long-term.

GOOD_RUBRIC = {
    "id": "explain-recursion-003",
    "input": {"user_message": "Explain recursion to a beginner"},
    "expected": {
        "scoring_rubric": [
            "Uses a concrete analogy (not just a technical definition)",
            "Mentions the base case explicitly",
            "Includes at least one short code example",
            "Does not exceed 200 words",
            "Avoids jargon like 'stack frame' without defining it first"
        ],
        "pass_threshold": 4  # must satisfy at least 4 of 5 criteria
    }
}

Bad rubric writing is vague ("explains it well"). Good rubric writing is checkable by a third party who has never seen the model's output, meaning each criterion is a yes/no question with an obvious answer once you read the response.

Step 4: Version the dataset like code

Treat the golden dataset as a first-class artifact in your repo, not a spreadsheet someone updates by hand and forgets to share. Store it as JSONL, one example per line, and put it under version control.

mkdir -p eval/golden
touch eval/golden/customer_support.jsonl
touch eval/golden/code_generation.jsonl
touch eval/golden/summarization.jsonl

Each category gets its own file. This keeps diffs readable in pull requests and lets teams own their slice independently. A typical PR that adds a regression case should look like a two-line JSON diff, nothing more.

import json

def append_example(filepath, example):
    with open(filepath, "a", encoding="utf-8") as f:
        f.write(json.dumps(example) + "\n")

def load_dataset(filepath):
    examples = []
    with open(filepath, encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if line:
                examples.append(json.loads(line))
    return examples

Tag every example with a semantic version bump in metadata whenever the underlying policy changes (refund window changes from 30 to 45 days, for instance). Do not silently edit an old example's expected field, since that erases the history of what the system used to be graded against. Instead bump metadata.version and note the change in the commit message.

Step 5: Build the eval runner

The runner loads the dataset, calls your model (or pipeline), and scores each response. Here is a minimal but complete version using an LLM-as-judge pattern for the rubric-based examples, alongside exact-match checks for the simpler ones.

import json
from dataclasses import dataclass
from typing import Optional

@dataclass
class EvalResult:
    example_id: str
    passed: bool
    score: float
    notes: str

def run_exact_checks(example, model_output):
    expected = example["expected"]
    text_lower = model_output.lower()

    must_contain = expected.get("must_contain", [])
    must_not_contain = expected.get("must_not_contain", [])

    missing = [p for p in must_contain if p.lower() not in text_lower]
    forbidden_hits = [p for p in must_not_contain if p.lower() in text_lower]

    passed = not missing and not forbidden_hits
    notes = ""
    if missing:
        notes += f"missing required phrases: {missing}. "
    if forbidden_hits:
        notes += f"contains forbidden phrases: {forbidden_hits}."

    return EvalResult(
        example_id=example["id"],
        passed=passed,
        score=1.0 if passed else 0.0,
        notes=notes.strip()
    )

def run_rubric_judge(example, model_output, judge_fn):
    rubric = example["expected"]["scoring_rubric"]
    threshold = example["expected"].get("pass_threshold", len(rubric))

    satisfied = 0
    notes = []
    for criterion in rubric:
        verdict = judge_fn(model_output, criterion)
        if verdict:
            satisfied += 1
        else:
            notes.append(f"failed: {criterion}")

    score = satisfied / len(rubric)
    passed = satisfied >= threshold

    return EvalResult(
        example_id=example["id"],
        passed=passed,
        score=round(score, 2),
        notes="; ".join(notes)
    )

def run_eval_suite(dataset_path, model_fn, judge_fn):
    results = []
    with open(dataset_path, encoding="utf-8") as f:
        for line in f:
            example = json.loads(line)
            model_output = model_fn(example["input"])

            if "scoring_rubric" in example["expected"]:
                result = run_rubric_judge(example, model_output, judge_fn)
            else:
                result = run_exact_checks(example, model_output)

            results.append(result)

    total = len(results)
    passed = sum(1 for r in results if r.passed)
    print(f"Passed {passed}/{total} ({passed / total:.0%})")

    for r in results:
        if not r.passed:
            print(f"  FAIL {r.example_id}: {r.notes}")

    return results

judge_fn is your LLM-as-judge call: pass the model output plus a single rubric criterion, ask the judge model for a strict yes/no with a one-sentence justification, and parse the verdict. Keep the judge prompt narrow (one criterion at a time) rather than asking it to grade five things at once. Narrow prompts produce far more consistent judge behavior.

Step 6: Wire it into CI

The whole point of a golden dataset is catching regressions before they ship. Add it as a gate in your CI pipeline so a prompt change or model swap cannot merge if it drops pass rate below a threshold.

name: llm-eval
on:
  pull_request:
    paths:
      - "prompts/**"
      - "eval/golden/**"
      - "src/pipeline/**"

jobs:
  run-golden-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        run: pip install -r eval/requirements.txt
      - name: Run golden dataset eval
        run: python eval/run_suite.py --min-pass-rate 0.90
        env:
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}

Set --min-pass-rate deliberately low at first (0.70 to 0.80) if your dataset is new, since a brand-new golden set often surfaces existing problems you didn't know you had. Ratchet it up over a few weeks as you fix real issues, not as you loosen rubrics to make the number look better.

Step 7: Keep the dataset alive

A golden dataset that stops growing goes stale within a quarter. Build these habits into the team's workflow:

Every bug report becomes an example. Before closing a ticket about a bad model response, add the exact input to the dataset with the correct expected block. This is the single highest-leverage habit for eval quality.
Review the dataset quarterly. Retire examples tied to deprecated features. Update rubrics when product policy changes.
Track pass rate over time, not just pass/fail per PR. A category (say, code_generation) trending down over three weeks even while individual PRs pass is an early warning sign worth investigating.
Rotate ownership. Assign each category file to one person. Ownerless datasets rot fastest.

A small script to track trend over time, appended to a results log after every CI run:

import json
import os
from datetime import datetime

def log_run(results, log_path="eval/history.jsonl"):
    total = len(results)
    passed = sum(1 for r in results if r.passed)
    entry = {
        "timestamp": datetime.now().isoformat(),
        "total": total,
        "passed": passed,
        "pass_rate": round(passed / total, 4),
        "failures": [r.example_id for r in results if not r.passed]
    }
    with open(log_path, "a", encoding="utf-8") as f:
        f.write(json.dumps(entry) + "\n")

def load_trend(log_path="eval/history.jsonl", last_n=10):
    if not os.path.exists(log_path):
        return []
    with open(log_path, encoding="utf-8") as f:
        lines = [json.loads(l) for l in f if l.strip()]
    return lines[-last_n:]

Chart pass_rate over timestamp in whatever dashboard your team already looks at daily. A dip is worth a Slack message before it is worth a postmortem.

FAQ

How many examples does a golden dataset need before it is useful? Twenty well-written examples per major task category is enough to catch obvious regressions. Quality of the expected block and diversity of edge cases matter far more than raw count. A hundred sloppy examples with vague rubrics are worse than twenty precise ones.

Should I use an LLM to generate the golden examples themselves? Use an LLM to draft candidate input fields and rough rubric ideas, but always have a human review and finalize the expected block. If the same model family generates both the questions and the grading criteria unsupervised, you inherit its blind spots directly into your ground truth.

How is a golden dataset different from a benchmark like MMLU? Public benchmarks measure general model capability and are useful for picking a base model. A golden dataset measures whether your specific product, with your specific prompts and retrieval pipeline, behaves correctly on your specific use cases. You need both, but only the golden dataset tells you if last night's prompt change broke refund handling.

What if my task is too subjective for exact-match checks? That is exactly what the rubric-based scoring in Step 3 is for. Break subjective quality into a checklist of concrete, checkable criteria (length, tone markers, required elements, forbidden elements) rather than trying to force a single exact string match on open-ended text.

How often should the pass-rate threshold in CI change? Raise it gradually as you fix real failures, and revisit it every time you add a meaningfully harder batch of examples (a pass rate can dip temporarily right after you add ten new adversarial cases, which is expected and fine). Never raise the threshold by deleting or softening rubrics just to hit a number.

Can the same golden dataset cover multiple models? Yes, and it should. Run the identical dataset against every candidate model or prompt version and compare pass rates side by side. This is how you make an evidence-based case for swapping models instead of relying on a handful of manual spot checks.