Ragas

Ragas Test Set Size: How Many Examples Do You Actually Need?

Pramod Dutta · Jun 2, 2026 · 15 min read

AUTHOR: Ira Menon

You built a RAG pipeline. Now what's a "big enough" test set?

Every team that starts evaluating a retrieval-augmented generation (RAG) system with Ragas eventually hits the same wall. They generate a handful of question-answer pairs, run the faithfulness and context precision metrics, get a score, and then someone in a standup asks the obvious question: "Is 20 examples even enough to trust this number?"

Nobody has a clean answer, so teams default to one of two bad habits. Either they ship with 15 examples because that's what fit in an afternoon, or they overcorrect and try to generate 2,000 synthetic examples because "more data is always better." Both are wrong, and both come from treating test set size as a vibe instead of a statistics problem.

This article walks through how to actually reason about Ragas test set size — what determines the right number, how to compute a rough minimum instead of guessing, how test set size interacts with metric choice, and where the size of your test set stops mattering and something else (like annotator agreement or knowledge base coverage) becomes the bottleneck. We'll write code along the way, because "how many examples" is not a philosophical question — it's a number you can actually calculate.

Why "how many examples" is really a statistics question in disguise

When you run Ragas on a test set, each metric — faithfulness, answer relevancy, context precision, context recall, and so on — produces a per-example score, and then Ragas (or you) averages those scores into a single number per metric. That averaged number is a sample statistic. It's an estimate of some "true" underlying quality of your pipeline, not the truth itself.

Any sample statistic has variance, and variance shrinks as sample size grows — but only up to a point, and only if your samples are representative of the traffic you actually care about. This is the same logic that governs A/B testing sample sizes, survey polling, and QA sampling in manufacturing. A test set of 10 examples can bounce your faithfulness score by 15-20 percentage points depending on which 10 examples you happened to pick. A test set of 300 well-distributed examples will barely move if you swap a handful out.

So the real question isn't "what's a magic number." It's: how much noise can you tolerate in your score, and how much does your underlying pipeline quality actually vary across different types of questions? Those two things — your tolerance for noise, and the natural variance in your data — are what determine test set size, not a rule of thumb someone posted on a forum.

Before reaching for a formula, it helps to separate the three variables that genuinely move the needle on test set size. Most guidance skips this and jumps straight to "use 50-100 examples," which is a fine rough default but doesn't tell you why, or when to deviate from it.

1. Score variance across your metrics. Some Ragas metrics are naturally noisier than others. Faithfulness (which checks if claims in the answer are actually supported by retrieved context) tends to be more binary and stable — an answer either hallucinates or it doesn't, so scores cluster near 0 or 1 with less middle ground. Answer relevancy, which uses an LLM judge to score how well the answer addresses the question, can have more spread because "relevance" is a fuzzier judgment call. Noisier metrics need larger samples to get a stable read.

2. The diversity of your query distribution. If your production traffic has five distinct query types — simple lookups, multi-hop questions, comparison questions, questions with no answer in the corpus, and ambiguous questions — your test set needs enough examples in *each* bucket to say something meaningful about that bucket. A 50-example test set that's 90% simple lookups tells you almost nothing about how your pipeline handles multi-hop reasoning, no matter how large the total number looks.

3. The decision you're making with the score. A test set for a quick sanity check during local development has a different bar than a test set used to gate a production deploy or compare two embedding models in a report to stakeholders. The tighter the decision, the tighter your confidence interval needs to be, which means more examples.

A rough formula for the minimum viable test set

You don't need a full power-analysis pipeline to get past guesswork. A standard error estimate gets you 80% of the way there. For a proportion-like metric (most Ragas metrics behave like proportions between 0 and 1), the standard error of the mean shrinks proportionally to the square root of your sample size.

Here's a small script that estimates the margin of error for a given test set size and score variance, so you can reason about it concretely instead of picking a number out of thin air:

import math

def margin_of_error(n, std_dev=0.25, confidence_z=1.96):
    """
    Estimate the margin of error for an average Ragas metric score.

    n: number of test examples
    std_dev: estimated standard deviation of per-example scores
             (0.25 is a reasonable default for LLM-judged metrics
             on a 0-1 scale; use 0.15 for more binary metrics
             like faithfulness, 0.30 for noisier ones)
    confidence_z: 1.96 for 95% confidence, 1.645 for 90%
    """
    standard_error = std_dev / math.sqrt(n)
    return confidence_z * standard_error


for n in [10, 20, 50, 100, 200, 300, 500]:
    moe = margin_of_error(n)
    print(f"n={n:>4}  margin of error ≈ ±{moe:.3f}")

Running this gives you a feel for the diminishing returns curve: going from 10 to 50 examples cuts your margin of error roughly in half, but going from 200 to 500 only shaves off a small additional slice. That's the core insight — the first 50-100 examples buy you the most precision per example added, and after that you're paying a lot of annotation and compute cost for smaller and smaller gains in confidence.

If a margin of error of ±0.10 is acceptable for your use case (i.e., you're fine saying "faithfulness is 0.82, plus or minus 10 points"), somewhere around 25-30 examples per query type might be enough. If you're trying to detect a 3-point regression between pipeline versions, you'll need several hundred examples, tight variance, or a paired comparison approach instead of comparing two independent averages.

It's worth being explicit about what this formula does and doesn't tell you. It assumes your examples are drawn independently and are reasonably representative of the population you care about — real user queries, not just whatever a synthetic generator happened to produce from the first few pages of your documentation. It also assumes the standard deviation estimate you plug in is roughly right, which you won't know for certain until you've actually run the metric on some real data. Treat the first pass as a starting estimate: run your metric on an initial batch of 30-40 examples, compute the actual standard deviation of the per-example scores, and then plug that real number back into the formula to get a better-calibrated target size. This two-pass approach — estimate, measure, re-estimate — beats guessing a single number upfront and living with it for the rest of the project.

It's also worth noting what the formula does *not* account for: correlated errors. If your retrieval component has a systematic blind spot — say, it consistently fails on questions about a specific product line because that product's documentation is chunked badly — no amount of independent random sampling fixes that. More examples from the same broken distribution just gives you a more confident measurement of a biased number. This is why stratification (covered next) matters as much as raw count. A statistically comfortable sample size on the wrong distribution is still the wrong answer.

Stratify before you scale — segment count beats total count

Here's the mistake most teams make once they internalize "bigger test set is better": they generate 300 examples from a synthetic data generator without checking what those 300 examples actually cover. Ragas' testset generation (via its knowledge-graph-based synthesizer) is good at producing plausible questions from your document corpus, but it will happily generate 280 simple single-hop questions and 20 multi-hop ones if that's the natural shape of your documents.

A better approach is to decide your segments first, then size each segment independently, then sum them up.

# Define the query types you actually care about in production
query_segments = {
    "simple_factual": 40,       # single-fact lookup
    "multi_hop": 40,             # requires combining 2+ chunks
    "comparison": 30,            # "how does X differ from Y"
    "out_of_scope": 20,          # correct answer is "not in the docs"
    "ambiguous_followup": 20,    # depends on conversation history
}

total_examples = sum(query_segments.values())
print(f"Total test set size: {total_examples}")
# -> 150, but distributed so each segment has enough
#    examples to report a segment-level score, not just
#    one blended number

This matters because a single blended Ragas score across an uneven distribution hides exactly the failure modes you built the eval to catch. If "out of scope" questions are only 5% of your test set, your pipeline can be terrible at admitting it doesn't know something and your overall context recall number will barely notice. Segment-level reporting, even on a modest total test set, tells you more than a large but undifferentiated one.

Using Ragas' testset generator without fooling yourself

Ragas ships a synthetic test set generator that builds a knowledge graph from your source documents and produces question-context-answer triples automatically. It's a huge time-saver over hand-writing every example, but it has a specific failure mode worth guarding against: it generates questions that are answerable from your documents by construction, because that's literally what the graph traversal is designed to do. That means your synthetic test set will systematically under-represent the "no good answer exists" case and edge cases that come from messy real user phrasing.

from ragas.testset import TestsetGenerator
from ragas.testset.graph import KnowledgeGraph
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

generator_llm = ChatOpenAI(model="gpt-4o-mini")
embeddings = OpenAIEmbeddings()

generator = TestsetGenerator(llm=generator_llm, embedding_model=embeddings)

# testset_size here is your target count -- but treat it as
# a starting point, not a finished product
testset = generator.generate_with_langchain_docs(
    documents=your_documents,
    testset_size=120,
)

df = testset.to_pandas()
print(df["synthesizer_name"].value_counts())

That last line is the important one. Ragas' generator tags each row with which "synthesizer" produced it (single-hop, multi-hop abstract, multi-hop specific, and so on depending on your configured query distribution). Check that value count before you trust the total. If you asked for 120 examples and got 100 single-hop and 20 multi-hop when you wanted an even split, adjust your query distribution config or top up manually rather than accepting whatever the default graph traversal handed you.

The practical workflow that works well: generate a larger synthetic pool (300-500), inspect the segment breakdown, then downsample to a curated set of 100-150 that actually matches your target distribution, and hand-write or manually source the 15-20 adversarial and out-of-scope examples that synthetic generation is bad at producing. That hybrid set beats a pure synthetic set of any size.

There's also a subtler issue with synthetic generation worth flagging: the questions it produces tend to be phrased the way the source documents are phrased, because the generator is literally reading chunks and building questions around them. Real users don't talk like your documentation. They misspell things, use informal shorthand, ask compound questions that mix two topics, and phrase things in ways that don't map cleanly onto any single chunk. If you only ever evaluate against synthetic questions, you're measuring how well your pipeline handles well-formed, documentation-flavored phrasing — which is a real and useful thing to measure, but it's not the same as measuring how your pipeline performs against actual production queries. If you have any query logs at all, even a small sample of 20-30 real historical queries mixed into your test set will surface failure modes that pure synthetic generation never will. Treat synthetic data as the bulk filler that gives you scale and coverage, and real query samples as the seasoning that keeps the test set honest.

Small test sets aren't useless, they're just a different tool

None of this means a 15-example smoke test is worthless. It has a real job: catching an obviously broken pipeline before it wastes anyone's time. If you change your chunking strategy and faithfulness drops from 0.85 to 0.40 on a 15-example set, you don't need statistical rigor to know something broke — that's not noise, that's a fire alarm.

Where small test sets fail is the opposite case: when a change looks like a small improvement or a small regression. A shift from 0.80 to 0.83 on 15 examples is almost certainly noise. On 200 well-distributed examples, that same shift is probably real. So the right mental model is tiered:

Tier 1 — dev smoke test (10-20 examples): run on every code change, catches catastrophic regressions, fast and cheap, not meant to detect small deltas.
Tier 2 — regression test set (100-150 examples, stratified): run before merging changes to retrieval, prompts, or chunking, meant to catch meaningful regressions in the 5-10 point range.
Tier 3 — release/benchmark test set (250-400+ examples, stratified, partially human-reviewed): run before major version comparisons, model swaps, or stakeholder reporting, meant to detect smaller shifts and support real conclusions.

Most teams only need Tier 1 and Tier 2. Tier 3 is worth building once your RAG system is customer-facing and a false "improvement" or false "regression" has real cost.

LLM-judge variance is a hidden multiplier on your required size

There's a wrinkle specific to Ragas and any LLM-as-judge evaluation framework: part of the noise in your score isn't just sampling variance from which examples you picked, it's judge variance — the same example scored twice by the judge LLM can get slightly different scores, especially at non-zero temperature or when the judge model itself is inconsistent on borderline cases.

This means your effective noise floor is higher than a pure human-labeled dataset of the same size would give you. You can quantify this cheaply:

from ragas import evaluate
from ragas.metrics import faithfulness
from datasets import Dataset

# Run the same small slice of the test set through Ragas
# multiple times to estimate judge-level noise, independent
# of which examples you picked
def judge_noise_check(dataset: Dataset, metric, runs=3):
    scores = []
    for i in range(runs):
        result = evaluate(dataset, metrics=[metric])
        scores.append(result[metric.name])
        print(f"Run {i + 1}: {metric.name} = {result[metric.name]:.4f}")

    spread = max(scores) - min(scores)
    print(f"\nSpread across {runs} runs: {spread:.4f}")
    return scores

# judge_noise_check(small_fixed_subset, faithfulness, runs=3)

If you run this on a fixed 20-example subset three times and see faithfulness swing between 0.78 and 0.86, that 8-point spread is pure judge noise, and it sets a floor below which you cannot meaningfully distinguish two pipeline versions no matter how large your test set is. In that case, increasing test set size helps average out judge noise across more examples, but a cheaper fix is often lowering judge temperature to 0, using a more deterministic evaluator model, or averaging multiple judge calls per example.

The practical implication for sizing is this: run the judge-noise check once, early, before you invest heavily in building out a large curated test set. If judge noise turns out to be small relative to the differences you're trying to detect, you can lean on the sample-size math from earlier with confidence. If judge noise turns out to be large, no amount of additional examples fully compensates — you need to fix the judge setup first, and only then does adding more examples start paying off in a predictable way. Teams sometimes skip this step and spend a week debating whether to grow their test set from 150 to 400 examples when the actual problem was a judge model running at default temperature 0.7, introducing more randomness into each individual score than the sample-size increase could ever average out.

A practical sizing checklist you can actually use

Pulling this together into something you can apply this week, without running a full statistical study:

List your query segments first. Simple lookup, multi-hop, comparison, out-of-scope, follow-up — whatever matches your actual traffic. Don't size the total until you know the buckets.
Aim for 20-30 examples per segment as a baseline, not 20-30 total. A five-segment breakdown at 25 each gives you 125 total, which is a reasonable Tier 2 regression set.
Check your synthesizer distribution if you're using Ragas' testset generator — don't trust the total count until you've verified the segment breakdown matches what you configured.
Run a judge-noise check on a fixed subset before deciding your test set is too small. Sometimes the bottleneck is judge consistency, not sample count, and no amount of added examples fixes that.
Match test set size to the decision. Smoke test: 10-20. Pre-merge regression gate: 100-150. Release benchmark or model comparison: 250-400+.
Revisit size when your traffic shifts. A test set built for last quarter's query patterns silently goes stale as user behavior changes — treat it as a living asset, not a one-time deliverable.
Prefer a smaller, well-labeled, stratified set over a larger, unreviewed synthetic dump. A hundred examples someone actually checked beats four hundred nobody has read.

None of this replaces judgment. A support-bot RAG system answering FAQ-style questions from a small, stable knowledge base can get away with a smaller, less stratified test set than a legal or medical RAG system where the cost of a wrong answer is high and the query space is huge. Size your test set to the cost of being wrong, not to a number you saw in a blog post.

Where this fits if you're building this for real

If you're evaluating a RAG pipeline for anything beyond a weekend project, the test set questions in this article — how many examples, how to stratify, how to separate judge noise from sampling noise — come up early and stay relevant through every iteration of your pipeline. Getting comfortable with Ragas metrics, testset generation, and how to read evaluation results without over- or under-trusting them is exactly what the Ragas Tutorial course on teachyou.ai walks through step by step, with working code for each stage: generating a synthetic test set, stratifying it by query type, running the core metrics, and interpreting scores with enough statistical grounding that you're not just staring at a number and guessing what it means.