Synthetic Test Set Generation with Ragas: A Practical Walkthrough

You shipped a RAG pipeline, wrote fifteen eval questions by hand, and it passed. Then a user asked something about a document you forgot existed, got a confidently wrong answer, and now you're debugging in production instead of in a test suite. This is the default failure mode for RAG evaluation, and it isn't a tooling problem — it's a coverage problem. Fifteen hand-written questions cannot represent a knowledge base with thousands of chunks across dozens of topics. You need hundreds of test cases, they need to touch every corner of your corpus, and you need them without spending three weeks writing them by hand. That's what synthetic test set generation with Ragas is for.

This walkthrough covers why manual eval sets don't scale, how Ragas actually generates synthetic question-answer pairs under the hood, the different query types you should be generating, and — the part almost everyone skips — how to filter synthetic data before you trust a single number it produces.

Why manual eval sets don't scale

Writing eval questions by hand feels rigorous. You read a document, you write a question about it, you write down what a good answer looks like. The problem is what this process implicitly optimizes for: the documents you remember, the topics you find interesting, and the phrasing that feels natural to you as the person who built the system.

Real users don't work that way. They ask about the boring appendix nobody re-reads. They phrase things awkwardly. They ask questions that require pulling together two unrelated sections. They ask questions your corpus can't actually answer, and your system needs to say so instead of hallucinating.

A RAG test set built entirely by hand tends to cluster around a handful of "obvious" documents and a narrow band of question difficulty — mostly simple factual lookups, because those are the easiest to write and grade. That gives you a false sense of confidence: your retrieval and generation scores look great on the eval set and then degrade the moment real traffic hits a part of the corpus nobody wrote a test for.

The scale problem compounds this. If your knowledge base has 500 documents split into 5,000 chunks, meaningful coverage means testing against a representative sample of those chunks — not the 20 you happened to think of. Manually writing hundreds of well-formed question-answer pairs, each requiring you to read the source material, formulate a natural question, and verify the answer, is not a weekend project. It's not even a sustainable ongoing process, and it needs to be ongoing because your corpus changes.

This is exactly the gap synthetic generation closes: mechanical, repeatable coverage across your entire document set, generated in the time it takes to run a script instead of the time it takes to run a workshop.

What a synthetic test set actually is

A synthetic test set for RAG evaluation is a collection of (question, reference_answer, reference_context) triples generated automatically from your own document corpus, rather than written by a human. The "reference" pieces matter — they're what let you later score a RAG pipeline's actual output against a known-good answer and known-relevant source chunks, using metrics like faithfulness, answer relevancy, and context precision.

Ragas builds these test sets by treating your documents as raw material for a generation pipeline, not by inventing questions from nothing. The output looks like this conceptually:

question: "What happens to a customer's subscription if a payment fails twice in a row?"
reference_context: [chunk_142, chunk_143]
reference_answer: "After two consecutive failed payment attempts, the subscription
enters a grace period of 7 days before automatic suspension..."
question_type: "simple"

The point isn't that this is a clever trick — it's that it's mechanical enough to run at scale. You can generate 20 of these or 2,000 of these with the same amount of manual effort, because the effort shifts from writing to reviewing.

How synthetic generation works conceptually

Under the hood, Ragas-style synthetic generation is a pipeline, not a single API call. Understanding the stages matters because each stage is a place quality can leak in — and a place you can intervene.

1. Chunk sampling. The pipeline starts from your existing document chunks (the same chunks your retriever indexes, ideally). It samples chunks to use as source material — sometimes single chunks, sometimes small clusters of related chunks pulled together to support harder question types.

2. Knowledge graph / relationship building. More sophisticated generation doesn't just look at one chunk in isolation. It builds relationships between chunks — which ones share entities, which ones are topically adjacent, which ones come from the same document versus different documents. This is what makes multi-hop question generation possible later: the pipeline needs to know which chunk pairs actually relate to each other before it can write a question that spans both.

3. Question synthesis. For a sampled chunk (or chunk cluster), an LLM is prompted to generate a question that chunk could plausibly answer, along with a reference answer grounded in that chunk's content. The prompt constrains the LLM to stay faithful to the source material — the goal is a question a real user might ask, answerable from what's actually in the corpus.

4. Style and persona variation. Real user populations aren't monolithic. Some ask terse, keyword-like queries. Some write full sentences. Some are domain experts using precise terminology, others are new to the topic and vague. Good synthetic generation varies phrasing style and can be pointed at specific personas to keep the test set from converging on one query style.

5. Evolution / complexity transformation. A first-pass question is often simple and factual. The pipeline can then "evolve" it — rewriting it to require reasoning across multiple chunks, adding a condition, or making it more indirect — rather than generating every complexity level from scratch each time.

The output of all this is a raw synthetic test set. Emphasis on raw — this is the input to a review step, not the finished product.

The three question types you actually need

Coverage isn't just about which documents you sample from — it's about which cognitive demands you place on the RAG pipeline. A test set made entirely of simple lookups will tell you your retriever works when the answer sits in one chunk with obvious keyword overlap. It won't tell you anything about the failure modes that actually hurt users in production.

Simple factual questions are direct lookups answerable from a single chunk: "What is the maximum file size for uploads on the free tier?" These are useful as a baseline sanity check — if your pipeline fails these, something is fundamentally broken in retrieval or the generator is ignoring context. But a test set that's 90% this type is not testing much.

Multi-hop reasoning questions require pulling information from two or more chunks, often from different sections or even different source documents, and combining them: "Is the refund policy the same for customers on the annual plan as it is for customers who upgraded mid-cycle?" Answering this correctly requires the retriever to surface both the refund policy chunk and the mid-cycle upgrade chunk, and requires the generator to actually synthesize across them rather than just paraphrasing whichever chunk came back first. This is where a lot of RAG pipelines quietly fail — retrieval that looks fine on single-hop questions often only fetches one of the two needed chunks here.

Conditional questions embed a constraint the system has to correctly apply: "What's the cancellation process if the account was created before the 2024 pricing change?" These test whether the system respects qualifiers instead of giving a generic answer that ignores the condition entirely. They're also a good proxy for how the system handles questions it should partially decline — if the corpus doesn't actually specify different behavior for the pre-2024 case, does the system fabricate a distinction, or does it correctly say the policy is the same regardless?

A reasonable starting mix, if you don't have a strong prior from your own traffic, is roughly half simple factual, a third multi-hop, and the remainder conditional — then adjust once real usage data tells you what your users actually ask.

A conceptual generation call

Here's what a synthetic generation step looks like conceptually, stripped down to the shape of the call rather than tied to one specific library version, since APIs here move fast:

from ragas.testset import TestsetGenerator
from ragas.testset.synthesizers import (
    SingleHopSpecificQuerySynthesizer,
    MultiHopAbstractQuerySynthesizer,
)

generator = TestsetGenerator(
    llm=generator_llm,
    embedding_model=embedding_model,
)

# distribution controls the mix of question types
query_distribution = [
    (SingleHopSpecificQuerySynthesizer(llm=generator_llm), 0.5),
    (MultiHopAbstractQuerySynthesizer(llm=generator_llm), 0.3),
    # a conditional/reasoning synthesizer would fill the remaining share
]

testset = generator.generate_with_langchain_docs(
    documents=corpus_documents,
    testset_size=300,
    query_distribution=query_distribution,
)

df = testset.to_pandas()

The important detail isn't the exact class names — those will shift across versions — it's the two things you control: the corpus you feed in, and the distribution across question types. Both are levers you should be deliberately tuning, not defaults you accept.

Also worth noting: generation isn't free or instant. Each question typically costs at least one LLM call, sometimes more for multi-hop questions that need relationship analysis first. Generating 300 questions across a large corpus is a batch job, not something you re-run casually on every commit — budget for it accordingly and treat your test set as a versioned artifact, not a build step.

Why you cannot skip the review step

This is the part of synthetic test set generation that gets skipped constantly, and it's the part that determines whether your eval numbers mean anything.

Synthetic questions have characteristic failure modes. An LLM generating a question from a chunk will sometimes produce something a real person would never ask — overly formal, oddly specific to phrasing that only exists in the source text, or referencing the document in a self-aware way ("According to the passage, what...") that no user query would ever contain. Some generated questions are trivially answerable by keyword match alone, which inflates retrieval metrics without testing anything meaningful about ranking quality. Others are subtly wrong: the "reference answer" doesn't actually follow from the cited context, usually because the source chunks were ambiguous or the LLM filled a gap with a plausible-sounding guess.

If you skip review and feed raw synthetic output straight into your evaluation pipeline, you get a number — a faithfulness score, a context precision score — that feels authoritative but is measuring against a flawed ground truth. A low score might mean your RAG pipeline is broken, or it might mean the synthetic reference answer was wrong. You can't tell which without having looked at the data. Worse, teams that trust unreviewed synthetic scores tend to optimize toward them, which means tuning a pipeline to satisfy artifacts of the generation process rather than real user needs.

Review doesn't have to mean reading every row by hand, though for anything under a few hundred examples that's honestly the most reliable option. At minimum, build automated filters for the failure patterns you can catch mechanically, and reserve human judgment for what's left.

A filtering step worth writing

Here's a conceptual filtering pass that catches the most common synthetic data problems before a human ever looks at the set:

def filter_synthetic_row(row):
    question = row["question"]
    answer = row["reference_answer"]
    context = row["reference_context"]

    # 1. reject questions that leak the fact they were generated from a document
    leaky_phrases = ["according to the passage", "based on the text",
                      "in the document", "as mentioned above"]
    if any(p in question.lower() for p in leaky_phrases):
        return False, "document-aware phrasing"

    # 2. reject answers that aren't actually grounded in the cited context
    grounding_score = check_answer_grounded_in_context(answer, context)
    if grounding_score < GROUNDING_THRESHOLD:
        return False, "answer not supported by cited context"

    # 3. reject questions that are trivially answerable by keyword overlap alone
    #    (a proxy for "this doesn't really test retrieval or reasoning")
    if keyword_overlap_ratio(question, context) > TRIVIAL_THRESHOLD:
        return False, "trivial keyword-match question"

    # 4. reject near-duplicate questions clustering on the same chunk
    if is_near_duplicate(question, seen_questions):
        return False, "duplicate coverage"

    return True, None

reviewed = [r for r in synthetic_rows if filter_synthetic_row(r)[0]]

The check_answer_grounded_in_context step is doing the heaviest lifting here — in practice this is often another LLM call asking "is this answer fully supported by this context, yes or no, and why," which is essentially the same faithfulness check you'll later run on your actual RAG pipeline's outputs, just applied one layer earlier to sanity-check the ground truth itself.

After automated filtering, sample the survivors — not all of them, but enough to trust the batch, maybe 15-20% for a large set — and read them as a human. You're looking for the things machines are bad at catching: questions that are technically well-formed but weird, answers that are correct but oddly specific in a way that would never generalize, and coverage gaps where an entire topic got skipped because your chunk sampling happened not to touch it.

Blending synthetic data with real user questions

Synthetic data solves coverage. It does not solve authenticity. Even a well-tuned generation pipeline produces questions that are recognizably "generated" in aggregate — a certain sameness of structure, a lack of the typos, abbreviations, and odd framing that real users bring. If you have any access to real user queries — support tickets, logged questions from a beta, a feedback form — that data is disproportionately valuable precisely because it's small and hard to fake.

The practical pattern is to treat these as complementary layers rather than competing sources:

• Synthetic data provides breadth. It's how you get systematic coverage across every document and chunk in your corpus, including the ones nobody would think to write a question about manually.
• Real user questions provide authenticity and reveal blind spots. They surface the actual phrasing, actual confusions, and actual multi-part questions your users bring — including questions about things you didn't realize were in your corpus, or things users assume are in your corpus but aren't.
• Real questions validate your synthetic generation itself. If you notice a category of real user question that your synthetic set never produces — a certain phrasing style, a certain kind of comparison, a certain edge case — that's a signal to adjust your synthesizer distribution or persona configuration, not just a one-off gap to patch.

A workable ratio for many teams is a synthetic set in the hundreds, providing the bulk of corpus coverage, supplemented by whatever real questions you can ethically and practically collect — even fifty real questions mixed in meaningfully changes what your eval catches, because they stress-test assumptions the synthetic generator doesn't know to question. Keep the two pools labeled separately in your results too; if your pipeline scores noticeably worse on the real-question subset than the synthetic subset, that gap itself is a useful signal about where synthetic coverage is systematically too easy.

One more practical note: treat both pools as living artifacts. As your corpus changes — new documents added, old ones deprecated — regenerate or extend the synthetic portion. As you collect more real traffic, periodically fold a fresh sample of real questions in and retire stale ones. An eval set frozen at launch tells you less and less about your system over time.

Putting it together

The workflow that holds up in practice looks roughly like this: generate a large synthetic pool across your whole corpus with a deliberate mix of simple, multi-hop, and conditional questions; run automated filters for the mechanical failure modes — leaky phrasing, ungrounded answers, trivial questions, duplicates; hand-review a meaningful sample of what survives; layer in whatever real user questions you can get your hands on, labeled separately so you can compare performance across the two pools; and revisit the whole set periodically rather than treating it as a one-time deliverable.

None of this replaces good judgment about your specific domain — a legal RAG system and a customer support bot need very different question distributions, and only you know which failure modes matter most for your users. What synthetic generation buys you is the ability to apply that judgment at scale, across a whole corpus, instead of across the fifteen documents you happened to remember.

If you're newer to RAG evaluation generally and want the foundation this builds on — how retrieval and generation fit together, what actually goes wrong in production RAG systems, and where evaluation fits into the development loop — that's exactly the ground we cover in Introduction to RAG.