DeepEval

DeepEval Synthetic Data Generation: Building Test Sets Automatically

Pramod Dutta · Jun 11, 2026 · 12 min read

The test data problem nobody talks about

Every team building an LLM application eventually hits the same wall: you know you need to evaluate your RAG pipeline or chatbot, but you don't have a test set. Writing test cases by hand is slow, biased toward whatever scenarios you happen to think of, and it never scales past fifty or so examples before the team gives up and ships without proper coverage. You end up evaluating your app against the five questions you tested manually during development, which tells you almost nothing about how it behaves on the long tail of real user queries.

This is exactly the gap DeepEval's Synthesizer was built to close. Instead of hand-writing goldens (DeepEval's term for a labeled test case), you point the Synthesizer at your source documents, your existing contexts, or even just a handful of example inputs, and it generates a structured evaluation dataset for you. The generated goldens are not just paraphrased questions — DeepEval's synthesis pipeline is built around evolutions, a technique that takes a straightforward question and evolves it into something harder: multi-hop reasoning, added constraints, comparative questions, or adversarial phrasing. That matters because a test set full of easy, single-fact questions will make a mediocre RAG pipeline look great, right up until it meets a real user.

In this article we'll walk through the actual Synthesizer API: generating goldens from documents, from raw contexts, from existing goldens, configuring evolution depth and styling, and finally turning synthetic goldens into a real evaluation run with metrics. By the end you'll have a repeatable pattern for bootstrapping test sets for any LLM application, RAG or otherwise.

Why synthetic data beats manual test-case writing

Before jumping into code, it's worth being explicit about why synthetic generation is the better default, not just a shortcut.

Coverage over documents, not intuition. When you write test cases by hand, you write questions about the parts of the document you remember. The Synthesizer chunks your source material and generates questions grounded in each chunk, so obscure sections of a policy document or a rarely-read part of your knowledge base get tested too.
Difficulty is controllable. DeepEval's evolution mechanism lets you dial up complexity — reasoning chains, hypothetical scenarios, comparative questions — rather than being stuck with whatever difficulty a human happened to write.
Consistent structure. Every synthetic golden comes out with the same shape: input, expected_output, and context. That consistency makes it trivial to feed straight into evaluate() without reformatting.
Speed. A few hundred goldens from a folder of PDFs takes minutes, not a week of an SME's time.
Repeatability. Because generation is driven by an LLM and a config object, you can regenerate a fresh, differently-worded dataset any time your source documents change, instead of maintaining a stale hand-written set.

None of this replaces human review entirely — you should still spot-check generated goldens before trusting them in CI — but it turns "we have no test set" into "we have a draft test set in twenty minutes," which is a much better starting position.

Installing and setting up DeepEval

Get the library installed and make sure your model provider key is available, since the Synthesizer uses an LLM internally to generate questions, answers, and evolutions.

pip install -U deepeval

# Synthesizer uses an LLM under the hood for generation and evolution
export OPENAI_API_KEY="sk-..."

DeepEval defaults to OpenAI models for synthesis unless you configure a custom model, exactly like it does for metrics. If your organization already wraps a different provider, you can pass a custom model into the Synthesizer constructor the same way you would for a metric — we'll touch on that later.

Generating goldens directly from documents

The most common entry point is generate_goldens_from_docs, which takes your raw source files — PDFs, .txt, .docx — chunks them, and produces goldens grounded in those chunks. This is the natural fit for RAG evaluation: you feed it the exact documents your retriever indexes, so the synthetic questions match the domain your app actually needs to answer.

from deepeval.synthesizer import Synthesizer
from deepeval.synthesizer.config import StylingConfig

styling_config = StylingConfig(
    input_format="Questions asked by customers of a SaaS billing platform",
    expected_output_format="A concise, accurate answer grounded strictly in the provided context",
    task="Answering customer support questions about invoices, refunds, and subscription changes",
    scenario="A customer messaging support through a chat widget",
)

synthesizer = Synthesizer(styling_config=styling_config)

synthesizer.generate_goldens_from_docs(
    document_paths=["docs/billing_policy.pdf", "docs/refund_faq.docx"],
    max_goldens_per_context=2,
)

# Each golden has .input, .expected_output, and .context populated
for golden in synthesizer.synthetic_goldens:
    print(golden.input)
    print(golden.expected_output)
    print(golden.context)
    print("---")

A few details worth understanding here:

`document_paths` accepts a list of file paths. DeepEval handles the chunking internally — you don't need to pre-split the documents yourself.
`max_goldens_per_context` controls how many question-answer pairs get generated per chunk. Push this higher for a bigger dataset, but each chunk only has so much unique information, so past a certain point you're generating near-duplicate questions.
`StylingConfig` is what keeps the generated data on-brand for your use case. Without it, the Synthesizer produces generic trivia-style questions. With it, the input_format, expected_output_format, task, and scenario fields steer the LLM toward the tone and structure your actual users would produce.

Generating goldens from raw contexts

Sometimes you don't have files sitting on disk — you have context strings already extracted from a database, a vector store dump, or a prior pipeline run. generate_goldens_from_contexts skips the document-chunking step entirely and works directly on lists of context strings you supply.

from deepeval.synthesizer import Synthesizer

synthesizer = Synthesizer()

contexts = [
    ["Refunds are processed within 5-7 business days to the original payment method."],
    ["Annual plans can be downgraded to monthly plans only at the end of the current billing cycle."],
    ["API rate limits reset every 60 seconds and are enforced per API key, not per account."],
]

synthesizer.generate_goldens_from_contexts(
    contexts=contexts,
    max_goldens_per_context=3,
)

print(f"Generated {len(synthesizer.synthetic_goldens)} goldens")

Notice that contexts is a list of lists — each inner list is a group of context strings that together should be sufficient to answer whatever question gets generated from them. This mirrors how retrieval actually works: your RAG pipeline usually pulls back multiple chunks per query, not just one, so grouping context strings this way keeps your synthetic data realistic.

This is also the API you'd reach for if you want to combine synthetic generation with contexts you already curated by hand — maybe you pulled the trickiest edge cases from your vector database and now want DeepEval to generate several phrasings of questions against each one.

Controlling evolution depth and style

The evolution step is what separates DeepEval's synthesis from naive "ask an LLM to write ten questions about this text" scripts. Evolutions rewrite an initial straightforward question into a harder variant, and DeepEval ships several evolution types out of the box, including reasoning, multi-context, concretizing, constraint-adding, comparative, and hypothetical-scenario evolutions.

You control this through EvolutionConfig:

from deepeval.synthesizer import Synthesizer
from deepeval.synthesizer.config import EvolutionConfig, Evolution

evolution_config = EvolutionConfig(
    evolutions={
        Evolution.REASONING: 1,
        Evolution.MULTICONTEXT: 1,
        Evolution.COMPARATIVE: 1,
        Evolution.HYPOTHETICAL: 1,
    },
    num_evolutions=2,
)

synthesizer = Synthesizer(evolution_config=evolution_config)

synthesizer.generate_goldens_from_docs(
    document_paths=["docs/billing_policy.pdf"],
    max_goldens_per_context=2,
)

The evolutions dictionary is a weighted map: DeepEval samples from the evolution types you list according to their relative weights, so setting them all to 1 gives an equal chance of each. num_evolutions controls how many evolution passes get applied to each base question — each pass compounds on the previous one, so num_evolutions=2 with REASONING and COMPARATIVE in the mix might produce a question that requires comparing two policies and reasoning about which one applies to a given scenario.

This is the lever to pull when your synthetic dataset feels too easy. If your RAG pipeline scores near-perfect faithfulness and answer relevancy on a first pass, that's often a signal the questions are too simple, not that the pipeline is flawless. Turning up num_evolutions or leaning on MULTICONTEXT and REASONING evolutions tends to expose retrieval gaps that simple factual questions never surface.

Expanding goldens you already have

Not every dataset starts from scratch. If you've already got a handful of hand-written goldens — maybe from a previous QA pass, a support-ticket export, or a small pilot set — you can use them as seeds and have the Synthesizer expand around them.

from deepeval.synthesizer import Synthesizer
from deepeval.dataset import Golden

seed_goldens = [
    Golden(
        input="How long does a refund take to process?",
        expected_output="Refunds are processed within 5-7 business days to the original payment method.",
    ),
    Golden(
        input="Can I switch from annual to monthly billing anytime?",
        expected_output="Downgrades from annual to monthly are only applied at the end of the current billing cycle.",
    ),
]

synthesizer = Synthesizer()

synthesizer.generate_goldens_from_goldens(
    goldens=seed_goldens,
    max_goldens_per_golden=3,
)

for golden in synthesizer.synthetic_goldens:
    print(golden.input)

This is a good pattern when a support or product team has already identified the questions that matter most, but you want more coverage around each one — different phrasings, related edge cases, harder variants — without asking a human to write all of them by hand.

From synthetic goldens to a real evaluation

Generated goldens are only useful once they're actually run through your application and scored. The synthetic_goldens list plugs directly into DeepEval's EvaluationDataset and then into evaluate(), alongside whichever metrics matter for your use case — typically AnswerRelevancyMetric, FaithfulnessMetric, and ContextualPrecisionMetric for a RAG pipeline.

from deepeval import evaluate
from deepeval.dataset import EvaluationDataset
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

dataset = EvaluationDataset(goldens=synthesizer.synthetic_goldens)

test_cases = []
for golden in dataset.goldens:
    actual_output, retrieved_context = my_rag_pipeline(golden.input)  # your app
    test_cases.append(
        LLMTestCase(
            input=golden.input,
            actual_output=actual_output,
            expected_output=golden.expected_output,
            retrieval_context=retrieved_context,
        )
    )

evaluate(
    test_cases=test_cases,
    metrics=[AnswerRelevancyMetric(), FaithfulnessMetric()],
)

The important step here is the loop in the middle: DeepEval generates the input and expected_output, but it's your job to actually run each golden.input through your real application to get actual_output and retrieved_context. The synthesizer doesn't know anything about your production pipeline — it only knows the source documents you fed it. This separation is deliberate. It keeps synthetic data generation decoupled from evaluation, so you can regenerate a dataset without touching your evaluation harness, or swap pipelines without touching the dataset.

Saving, versioning, and reusing synthetic datasets

A synthetic dataset you generate once and never look at again isn't worth much more than a one-off manual test. Treat it like any other test asset: save it, check it into version control (or push it to Confident AI if your team uses DeepEval's hosted platform), and re-run the same evaluation as your prompts and models change over time.

# Save to disk as CSV or JSON for version control
synthesizer.save_as(
    file_type="json",
    directory="./test-datasets",
)

A practical workflow that works well for most teams:

Generate an initial synthetic dataset from your current documents.
Manually review a sample — DeepEval's evolutions are good, but not infallible, and occasionally produce a question that doesn't quite match its expected_output. Spot check 10-20% of goldens before trusting the set.
Save the reviewed dataset and commit it alongside your evaluation code.
Re-run evaluate() against this fixed dataset every time you change a prompt, swap a model, or update your retriever, so you're comparing apples to apples across changes.
Periodically regenerate a fresh synthetic batch when your source documents change meaningfully, and merge the best new goldens into your versioned set.

This turns synthetic generation from a one-time bootstrapping trick into an ongoing part of your evaluation pipeline, rather than something you run once and forget about.

Common pitfalls when generating synthetic data

A few things trip people up the first time they use the Synthesizer, worth calling out directly:

Skipping `StylingConfig` and getting generic questions. Without a styling config, generated inputs default to a neutral, general-purpose tone that often doesn't match how your actual users phrase things. Always set input_format and scenario to match your real user base.
Setting `max_goldens_per_context` too high on small documents. If a document chunk only contains one or two facts, asking for five goldens per context produces redundant or forced questions. Start conservative — two or three per context — and scale up on larger corpora.
Treating synthetic `expected_output` as ground truth without review. The expected output is itself LLM-generated from the context, so it can occasionally be subtly wrong or miss nuance a human expert would catch, especially in specialized domains like legal or medical content. Review is not optional if the domain has regulatory stakes.
Never adjusting evolution settings. Leaving evolutions at default settings on every project produces datasets that don't stress the specific failure modes your app actually has. A pipeline with weak multi-document retrieval needs more MULTICONTEXT evolutions in its test set; a pipeline that struggles with edge-case constraints needs more constraint-based evolutions.
Forgetting that generation cost scales with dataset size. Every golden — and every evolution pass on top of it — is an LLM call. Generating a thousand highly-evolved goldens from a large document set is not free or instant; budget for it the same way you'd budget for any other LLM-heavy pipeline in CI.

Wrapping up

DeepEval's Synthesizer turns the "we don't have a test set" problem into a solvable, repeatable step in your evaluation pipeline. Whether you're generating from raw documents with generate_goldens_from_docs, working directly from context strings with generate_goldens_from_contexts, or expanding a small hand-curated seed set with generate_goldens_from_goldens, the output is the same consistent Golden structure that plugs straight into EvaluationDataset and evaluate(). Combine that with EvolutionConfig to control how hard the questions get, and StylingConfig to keep the generated data on-brand for your actual users, and you have a dataset-generation workflow that scales with your application instead of bottlenecking on however many test cases a human had time to write.

The real payoff shows up over time: a versioned, regenerable synthetic dataset means every prompt change, model swap, or retriever tweak gets evaluated against the same rigorous baseline, instead of against whatever five questions happened to be top of mind that day.

If you want a guided, hands-on walkthrough of the full DeepEval workflow — synthesizers, metrics, datasets, and CI integration — check out the DeepEval Tutorial course on teachyou.ai, where we build a complete evaluation pipeline for a real RAG application from scratch.