Building a Golden Dataset for RAG Evaluation

Most RAG evaluation starts and ends with "does this answer look reasonable?" That approach works until you change your chunking strategy, swap an embedding model, or add a reranker, and you have no way to tell whether the change helped or hurt. A rag golden dataset is the fix: a curated, versioned set of questions with known-correct answers and known-relevant source passages that you can run against your pipeline every time something changes. This article walks through how to build one from scratch, how big it needs to be, what fields it needs, and how to keep it from rotting.

Why ad hoc evaluation fails

Teams that skip golden datasets usually fall into one of three traps.

The first is "vibes-based" evaluation: someone asks the chatbot five questions, reads the answers, and says "looks good, ship it." This catches nothing beyond gross failures. It cannot detect a 10% drop in retrieval precision, and it definitely cannot tell you whether last week's chunk-size change broke answers about a specific product SKU that nobody happened to ask about during manual testing.

The second trap is using production logs as your only evaluation source. Logs are useful for discovering failure patterns, but they are unlabeled: you know what the system answered, not whether that answer was correct. Without ground truth, you cannot compute precision, recall, or faithfulness scores. You can only eyeball outputs, which brings you back to the vibes problem.

The third trap is over-relying on LLM-as-judge scores with no human-verified answers to calibrate against. An LLM judge grading its own pipeline's outputs, with no anchor to compare to, will happily rate a confidently wrong answer as "helpful and accurate." A rag golden dataset gives the judge (and you) something concrete to check against.

What a golden dataset actually contains

A golden dataset for RAG evaluation is not just question-answer pairs. Each entry needs enough structure to evaluate both halves of the pipeline separately: retrieval and generation.

At minimum, each row needs:

• question: the exact query text, written the way a real user would phrase it, not the way an engineer would.
• reference_answer: a human-verified, correct answer, written in the style you want the system to produce.
• relevant_chunk_ids: the specific document chunks (or passage IDs) that contain the information needed to answer correctly. This is what lets you score retrieval independently of generation.
• question_type: a category label (see below), because aggregate scores hide category-specific failures.
• difficulty: simple lookup, multi-hop, or requires-inference. Averaging these together masks where your pipeline actually struggles.
• metadata: source document version, date created, who verified it, and any notes on ambiguity.

Here is a minimal JSON schema for one entry:

{
  "id": "gs-0142",
  "question": "What is the maximum context window for the enterprise plan?",
  "reference_answer": "The enterprise plan supports a 200K token context window, configurable up to 1M tokens on request.",
  "relevant_chunk_ids": ["pricing-doc-v3::chunk-17", "pricing-doc-v3::chunk-18"],
  "question_type": "factual_lookup",
  "difficulty": "simple",
  "source_document": "pricing-doc-v3.pdf",
  "created_by": "human_review",
  "notes": "Answer changed in v3 revision, verify against latest pricing page before reuse"
}

Keep this in JSONL, one object per line. It is easy to diff in git, easy to append to, and every eval framework (Ragas, DeepEval, promptfoo, custom scripts) can load it without conversion.

Sourcing the questions

There are four legitimate sources for golden questions, and you want all four, not just one.

1. Domain expert authored. Pull in the person who actually answers these questions today: a support lead, a solutions engineer, a compliance officer. Ask them to write 20-30 questions they get asked weekly, along with the correct answer in their own words. This is your highest-quality, lowest-volume source. Budget a half day of their time per 25 questions including review.

2. Mined from real usage. Pull the top N most frequent queries from search logs or support tickets (with PII stripped). These are exactly the questions your system needs to nail, because they represent real demand, not hypothetical edge cases. You will need to write the reference answers yourself or route them to a domain expert for verification.

3. LLM-generated, human-verified. Have an LLM read your source documents chunk by chunk and generate candidate question-answer pairs, then have a human review and correct every single one before it enters the golden set. Never skip the human review step here: LLM-generated questions frequently ask about details that are wrong, ambiguous, or not actually present in the source. Treat LLM generation as a first draft, not ground truth.

A working prompt pattern for generation:

You are creating evaluation questions from a document chunk.
Read the chunk below and generate 2 questions a real user might ask
that can be answered using ONLY this chunk's content.

For each question, also write the correct answer using only
information present in the chunk. If the chunk does not contain
enough information for a good question, output NONE.

Chunk:
{chunk_text}

Output format (JSON list):
[{"question": "...", "answer": "..."}]

Run this over every chunk in a sample of your corpus, dedupe near-identical questions, and route the output to human review before anything is marked "golden."

4. Adversarial and negative cases. This is the category teams skip and the one that catches the most production bugs. You need:

• Questions with no answer in the corpus at all, where the correct system behavior is to say "I don't know" or "I don't have that information," not to hallucinate.
• Ambiguous questions that could refer to multiple documents (e.g., "what's the refund policy" when there are three refund policies for three product tiers).
• Questions that require combining information from two or more chunks (multi-hop), which single-vector retrieval often fails.
• Out-of-scope questions unrelated to your corpus, to verify the system declines gracefully instead of making something up.

Aim for these adversarial cases to be at least 20% of your total set. A rag golden dataset that only contains answerable, unambiguous, single-chunk questions will systematically overstate how well your system performs in production, where real users ask messy questions.

How big does it need to be

There is no universal number, but here are working guidelines by stage:

• Early development / smoke tests: 30-50 questions is enough to catch obvious breakage after a pipeline change. Run this on every commit or every retrieval-config change; it should take under a minute.
• Pre-launch evaluation: 150-300 questions, covering every question_type and difficulty bucket you care about, with at least 20-40 adversarial cases. This is your regression suite.
• Ongoing production monitoring: keep growing this set. Every time a user-reported failure comes in, turn it into a golden dataset entry after you fix it, so it never regresses silently again. This is the single highest-leverage habit for long-term RAG quality: treat every bug as an unwritten test case.

Stratify your set across question types so you can report per-category metrics, not just one aggregate number. A pipeline can score 85% overall while scoring 40% on multi-hop questions, and an aggregate score will hide that completely.

Separating retrieval evaluation from generation evaluation

This is the part most teams get wrong: they measure the final answer's quality and stop there. If the answer is wrong, you don't know if retrieval failed (wrong or missing chunks passed to the LLM) or generation failed (right chunks, but the LLM answered poorly anyway). Fixing the wrong half wastes a sprint.

Score retrieval with your relevant_chunk_ids field, independent of what the LLM eventually says:

• Recall@k: of the chunks marked relevant in your golden entry, what fraction appear in the top-k retrieved results? This tells you if your retriever is finding the right material at all.
• Precision@k: of the top-k retrieved chunks, what fraction are actually relevant? Low precision means you are stuffing the context window with noise, which degrades generation even when the right chunk is technically present.
• MRR (Mean Reciprocal Rank): how high up does the first relevant chunk rank? A relevant chunk buried at position 8 out of 10 often gets ignored by the LLM even if it's technically "retrieved."

Score generation separately, using the reference_answer as ground truth, once you know retrieval delivered the right context:

• Faithfulness: does the generated answer only state things supported by the retrieved chunks, with no fabricated details?
• Answer correctness: does the generated answer match the reference_answer in substance, not necessarily wording? This is where an LLM-as-judge, calibrated against your human-verified reference_answer, does the heavy lifting.
• Completeness: for multi-part questions, does the answer cover every part, or does it answer only the easiest sub-question and drop the rest?

A simple harness in Python, using your JSONL golden set:

import json

def load_golden_set(path):
    with open(path) as f:
        return [json.loads(line) for line in f]

def evaluate_retrieval(entry, retrieved_chunk_ids, k=5):
    relevant = set(entry["relevant_chunk_ids"])
    top_k = retrieved_chunk_ids[:k]
    hits = [c for c in top_k if c in relevant]
    recall = len(hits) / len(relevant) if relevant else None
    precision = len(hits) / len(top_k) if top_k else 0
    return {"recall_at_k": recall, "precision_at_k": precision}

def run_eval(golden_path, retriever_fn, generator_fn, judge_fn):
    results = []
    for entry in load_golden_set(golden_path):
        retrieved = retriever_fn(entry["question"])
        retrieval_scores = evaluate_retrieval(entry, retrieved)
        answer = generator_fn(entry["question"], retrieved)
        judge_score = judge_fn(entry["reference_answer"], answer)
        results.append({
            "id": entry["id"],
            "question_type": entry["question_type"],
            "retrieval": retrieval_scores,
            "generation_score": judge_score,
        })
    return results

Run this after every change to chunking, embedding model, retriever config, reranker, or prompt template, and diff the results against your last baseline run. Store baseline runs in version control alongside the golden set itself so regressions are visible in a pull request, not discovered by an angry user three weeks later.

Keeping the dataset from going stale

A golden dataset is only as good as its alignment with the current corpus. Two things break it silently:

Source documents change underneath it. If your pricing page updates and your golden entry still says the old number, your eval will start marking correct answers as wrong, and you'll waste time debugging a pipeline that isn't broken. Tag every entry with the source document version, and run a periodic check (monthly is reasonable for most teams) that flags any golden entry whose source document has since changed.

The corpus grows and old entries stop being representative. If you launched with 200 documents and now have 2,000, your original golden set is testing a fraction of what the system needs to handle. Refresh the set quarterly by mining new production queries and adding coverage for newly added document categories.

Version the golden set itself in git, same as code. Tag releases (golden-v1, golden-v2) so you can always reproduce a historical eval run against the exact dataset that produced it, and so you can see exactly which entries were added or corrected between releases.

A minimal build plan

If you are starting from zero, here is a realistic two-week plan for a small team:

1. Days 1-2: Pull top 100 queries from logs or support tickets. Draft reference answers using existing documentation.
2. Days 3-4: Run LLM-assisted question generation over your corpus, producing 3-4x more candidates than you need.
3. Days 5-7: Human review pass. A domain expert verifies or corrects every candidate answer and marks relevant_chunk_ids by hand. Reject anything ambiguous or unverifiable rather than forcing it in.
4. Day 8: Write 30-40 adversarial cases by hand: unanswerable questions, multi-hop questions, out-of-scope questions.
5. Days 9-10: Tag every entry with question_type and difficulty, load into JSONL, commit to version control, write the eval harness.
6. Days 11-14: Run the baseline eval, review per-category scores, and use the lowest-scoring category to prioritize your next pipeline fix.

From there, treat the dataset as a living artifact: every production bug report becomes a new entry after you fix the underlying issue, every quarter you refresh for corpus drift, and every pipeline change gets run against it before merging.

FAQ

How many questions do I need before a golden dataset is useful? Even 30-50 well-chosen questions, spanning simple lookups, multi-hop questions, and at least a few unanswerable cases, will catch the majority of regressions from pipeline changes. Start small and grow it continuously rather than waiting until you have "enough."

Can I use an LLM to generate the entire golden dataset without human review? No. LLM-generated questions and answers frequently contain errors, hallucinated details, or trivial questions that don't test anything meaningful. Use LLM generation to produce a first draft at volume, then have a human verify every entry before it counts as "golden." An unverified dataset gives you false confidence, which is worse than no dataset.

Should the golden dataset include questions my RAG system currently gets wrong? Yes, deliberately. The point of the dataset is to measure progress, not to flatter your current pipeline. Include known-hard cases so you can track whether changes actually improve them over time.

How is this different from using Ragas or DeepEval directly? Ragas, DeepEval, and similar frameworks are the scoring layer: they compute metrics like faithfulness, answer relevancy, and context precision. They still need a golden dataset with reference answers and relevant chunk IDs as input. The framework does not replace the work of curating good questions and verified answers; it consumes what you build here.

What's the difference between a golden dataset and a benchmark like a standard QA dataset? Public QA benchmarks test general language understanding and are not specific to your corpus, your documents, or your users' actual phrasing. A golden dataset is built from your own documents and your own real or realistic queries, so it measures whether your specific RAG pipeline works for your specific use case, not whether the underlying LLM is generally capable.

How often should I re-run the full evaluation? Run the smoke-test subset on every pipeline-affecting commit. Run the full regression suite before any release that touches retrieval, chunking, embeddings, or prompts. Refresh the dataset itself on a quarterly cadence, or immediately after a significant corpus update.