DeepEval Test Organization: Structuring a Large Eval Suite

The eval suite that nobody wants to touch

Every team that adopts DeepEval starts the same way: a single test_app.py, a handful of LLMTestCase objects, and a couple of metrics like AnswerRelevancyMetric and FaithfulnessMetric. It works. It's fast to write. Then six months pass. The file is 2,000 lines long, half the test cases have names like test_case_2_final_v2, nobody remembers which dataset a given assertion pulls from, and CI takes eleven minutes because every PR reruns every metric against every case regardless of what changed.

This is not a DeepEval problem. It is a test organization problem that happens to show up in eval suites because eval suites grow in a specific, sneaky way: not through deliberate feature work, but through accretion. Someone adds three test cases after a bad production incident. Someone else bolts on a hallucination check. A PM asks for a "tone" metric. None of these additions are wrong on their own, but without a structure to slot them into, they pile up into a suite that is expensive to run and impossible to reason about.

This article is about the structure — how to lay out directories, name test cases, group metrics, tag suites, and wire CI so that a DeepEval project with thousands of assertions is still navigable a year from now. If you're new to DeepEval itself, our DeepEval Tutorial course covers the fundamentals; this piece assumes you already know what LLMTestCase and assert_test do and want to know how to scale past the toy example.

Start from the shape of your application, not your metrics

The most common organizational mistake is structuring the test suite around metrics first — a test_faithfulness.py, a test_relevancy.py, a test_toxicity.py — instead of around the parts of the application under test. Metrics are reusable tools; they are not the unit of organization. Your application's components are.

A RAG chatbot, a summarization pipeline, and an agent with tool calls are different systems with different failure modes, and your directory tree should say so before you even open a file:

tests/
  eval/
    conftest.py
    datasets/
      retrieval_golden_set.jsonl
      support_tone_examples.jsonl
    fixtures/
      metrics.py
      test_data.py
    retrieval/
      test_chunk_relevance.py
      test_context_recall.py
    generation/
      test_faithfulness.py
      test_answer_relevancy.py
    agent/
      test_tool_correctness.py
      test_task_completion.py
    regression/
      test_known_failures.py

Notice that retrieval, generation, and agent map to actual components in the pipeline. When retrieval breaks, you know which folder to open. When someone changes the system prompt, they know generation/ is the folder that needs a rerun, not the entire suite. This mapping also makes ownership obvious — the person who owns the retriever owns retrieval/, and code review naturally routes to the right reviewer.

The regression/ folder deserves special mention. Every eval suite eventually accumulates specific inputs that broke production once. Keep those separate from your general-purpose golden set. They are not there to measure quality broadly; they are there to make sure a fixed bug stays fixed.

Test case granularity: one assertion, one intent

A second failure mode is cramming multiple unrelated checks into a single test function because it's convenient to reuse the same LLM call. Resist this. In DeepEval, each test_case should represent one intent, even if that means calling the same generation function twice across two tests.

from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_faithfulness_on_refund_policy_question(rag_pipeline):
    result = rag_pipeline.query("What is the refund window for annual plans?")
    test_case = LLMTestCase(
        input="What is the refund window for annual plans?",
        actual_output=result.answer,
        retrieval_context=result.retrieved_chunks,
    )
    metric = FaithfulnessMetric(threshold=0.8, model="gpt-4o-mini")
    assert_test(test_case, [metric])


def test_relevancy_on_refund_policy_question(rag_pipeline):
    result = rag_pipeline.query("What is the refund window for annual plans?")
    test_case = LLMTestCase(
        input="What is the refund window for annual plans?",
        actual_output=result.answer,
        retrieval_context=result.retrieved_chunks,
    )
    metric = AnswerRelevancyMetric(threshold=0.7, model="gpt-4o-mini")
    assert_test(test_case, [metric])

This looks redundant, and it is — you're paying for two LLM calls to the pipeline instead of one. But when test_faithfulness_on_refund_policy_question fails in CI, you know exactly what failed and why: faithfulness dropped, not relevancy, not both, not "something in the refund question test." If you bundle both metrics into one test with one assertion, a single failing metric among five kills the whole test, and your failure log tells you nothing about which of the five broke.

If duplicate LLM calls are a real cost concern (they usually are, at scale), don't merge the tests — cache the pipeline output instead. Use a fixture scoped to the test case content so the same input only invokes the pipeline once:

import pytest

@pytest.fixture(scope="module")
def refund_policy_result(rag_pipeline):
    return rag_pipeline.query("What is the refund window for annual plans?")


def test_faithfulness_on_refund_policy(refund_policy_result):
    test_case = LLMTestCase(
        input="What is the refund window for annual plans?",
        actual_output=refund_policy_result.answer,
        retrieval_context=refund_policy_result.retrieved_chunks,
    )
    assert_test(test_case, [FaithfulnessMetric(threshold=0.8)])


def test_relevancy_on_refund_policy(refund_policy_result):
    test_case = LLMTestCase(
        input="What is the refund window for annual plans?",
        actual_output=refund_policy_result.answer,
        retrieval_context=refund_policy_result.retrieved_chunks,
    )
    assert_test(test_case, [AnswerRelevancyMetric(threshold=0.7)])

You get isolated, single-intent assertions without paying for the pipeline call twice.

Naming conventions that survive a thousand test cases

A test suite is a filing system, and filing systems fail when names stop encoding information. Two conventions pay for themselves quickly:

• Name test functions after behavior, not implementation. test_answer_relevancy_metric_call tells you what code runs. test_rejects_answers_that_ignore_retrieved_context tells you what behavior is being guaranteed. When it fails, the second name is a bug report; the first is a stack trace.
• Name test cases inside a dataset by scenario id, not sequence number. refund_edge_case_partial_year survives reordering and merging datasets. case_014 does not — it silently shifts meaning the moment someone inserts a row above it.

For datasets stored as JSONL or CSV, add a stable id field independent of row order:

import json

def load_golden_set(path):
    cases = []
    with open(path) as f:
        for line in f:
            row = json.loads(line)
            assert "id" in row, f"missing id in row: {row}"
            cases.append(row)
    return cases

Enforcing the id field with an assertion at load time is cheap insurance. It's the difference between a dataset that can be safely split across two files later and one that can't be touched without breaking every reference to "case 14" in a Slack thread somewhere.

Grouping with tags and markers, not folder sprawl

DeepEval integrates with pytest, which means you get pytest's marker system for free — and markers are a better tool than deeply nested folders for cross-cutting concerns like "runs in every PR" versus "runs nightly" versus "expensive, runs weekly."

# conftest.py
import pytest

def pytest_configure(config):
    config.addinivalue_line("markers", "smoke: fast subset run on every PR")
    config.addinivalue_line("markers", "nightly: full regression run, scheduled")
    config.addinivalue_line("markers", "expensive: multi-turn or long-context evals")

import pytest
from deepeval import assert_test

@pytest.mark.smoke
def test_faithfulness_basic_question(rag_pipeline):
    ...

@pytest.mark.nightly
@pytest.mark.expensive
def test_faithfulness_across_full_golden_set(rag_pipeline, golden_set):
    ...

Then your CI config gets simple, tiered commands instead of one giant deepeval test run:

deepeval test run tests/eval -m smoke
deepeval test run tests/eval -m nightly --nightly-flag
deepeval test run tests/eval -m "expensive and not smoke"

This is the single highest-leverage change most teams can make to a slow eval suite. A PR does not need to know that the expensive multi-turn agent evals still pass — it needs to know that the fast, targeted subset relevant to the change still passes. Save the exhaustive run for a nightly cron job and let engineers get PR feedback in under two minutes instead of eleven.

Centralize metric configuration instead of scattering thresholds

If threshold=0.8 for FaithfulnessMetric is typed inline in forty different test files, changing your quality bar means forty edits and forty chances to miss one. Centralize metric construction in a fixtures module and import from there:

# fixtures/metrics.py
from deepeval.metrics import (
    FaithfulnessMetric,
    AnswerRelevancyMetric,
    ContextualPrecisionMetric,
)

DEFAULT_MODEL = "gpt-4o-mini"

def faithfulness_metric(threshold: float = 0.8) -> FaithfulnessMetric:
    return FaithfulnessMetric(threshold=threshold, model=DEFAULT_MODEL, include_reason=True)

def answer_relevancy_metric(threshold: float = 0.7) -> AnswerRelevancyMetric:
    return AnswerRelevancyMetric(threshold=threshold, model=DEFAULT_MODEL, include_reason=True)

def contextual_precision_metric(threshold: float = 0.75) -> ContextualPrecisionMetric:
    return ContextualPrecisionMetric(threshold=threshold, model=DEFAULT_MODEL)

from fixtures.metrics import faithfulness_metric, answer_relevancy_metric
from deepeval import assert_test
from deepeval.test_case import LLMTestCase

def test_faithfulness_on_billing_question(billing_result):
    test_case = LLMTestCase(
        input="When does my invoice get generated?",
        actual_output=billing_result.answer,
        retrieval_context=billing_result.retrieved_chunks,
    )
    assert_test(test_case, [faithfulness_metric()])

Now raising the global faithfulness bar from 0.8 to 0.85 for a product launch is a one-line change in fixtures/metrics.py, not a grep-and-replace across the repo. You also get a natural place to document *why* a threshold is what it is — a comment next to faithfulness_metric explaining "0.8 chosen after false-positive review in March" is worth more than the same comment scattered across forty call sites, where nobody will find it during the next threshold discussion.

Keep an explicit exceptions path too, for the rare test that genuinely needs a different bar:

def test_faithfulness_on_ambiguous_legal_question(legal_result):
    test_case = LLMTestCase(
        input="Can I terminate mid-contract without penalty?",
        actual_output=legal_result.answer,
        retrieval_context=legal_result.retrieved_chunks,
    )
    # Legal questions tolerate more hedging language; lower bar is intentional.
    assert_test(test_case, [faithfulness_metric(threshold=0.65)])

Separate golden datasets from test code

Test *logic* and test *data* age at different rates. Your assertions about what "good" looks like change rarely. Your example inputs — real user questions, edge cases discovered in production, adversarial prompts — grow continuously. Mixing them in the same Python file means every new example requires a code review of logic that hasn't changed, and it means datasets can't be reused across test files or shared with a non-engineering reviewer (a support lead who wants to add ten new tricky questions shouldn't need to read pytest).

Keep datasets in their own directory as JSONL, and write one small loader:

# fixtures/test_data.py
import json
from pathlib import Path

DATA_DIR = Path(__file__).parent.parent / "datasets"

def load_dataset(name: str) -> list[dict]:
    path = DATA_DIR / f"{name}.jsonl"
    with open(path) as f:
        return [json.loads(line) for line in f]

import pytest
from fixtures.test_data import load_dataset
from fixtures.metrics import faithfulness_metric
from deepeval import assert_test
from deepeval.test_case import LLMTestCase

@pytest.mark.nightly
@pytest.mark.parametrize("row", load_dataset("retrieval_golden_set"), ids=lambda r: r["id"])
def test_faithfulness_across_golden_set(row, rag_pipeline):
    result = rag_pipeline.query(row["question"])
    test_case = LLMTestCase(
        input=row["question"],
        actual_output=result.answer,
        retrieval_context=result.retrieved_chunks,
        expected_output=row.get("expected_answer"),
    )
    assert_test(test_case, [faithfulness_metric(row.get("threshold", 0.8))])

This single parametrized test now scales to however many rows are in retrieval_golden_set.jsonl without any new code. Adding a hundred new examples is a data change, reviewable by anyone who understands the domain, not a code change gated on engineering bandwidth.

Handling flaky and non-deterministic evals

LLM-as-judge metrics are inherently noisier than traditional assertions — the same input can score 0.79 one run and 0.82 the next. Left unmanaged, this noise erodes trust in the whole suite: engineers start ignoring red CI because "it's probably just the eval being flaky," which is the same trap as ignoring flaky unit tests, except worse, because it happens by default rather than by mistake.

Three concrete mitigations:

• Use `include_reason=True`, always. When a metric fails, you want the judge's explanation in the CI log immediately, not a bare score. This turns a five-minute "is this really broken?" investigation into a ten-second read.
• Set thresholds with margin, not at the edge of acceptable. If your actual quality bar is "faithfulness must never meaningfully drop," don't set threshold=0.75 if your typical passing score is 0.76. Give yourself room — 0.65 with a documented target of 0.8+ typical — so ordinary judge variance doesn't cause false alarms.
• Track score trends, not just pass/fail. DeepEval integrates with Confident AI for this, but even a simple CSV log of scores per test case per run, committed or shipped to a dashboard, lets you catch a metric drifting from 0.85 average to 0.78 average over two weeks — a real regression a binary pass/fail threshold would miss entirely until it finally crosses the line.

from deepeval.metrics import FaithfulnessMetric

metric = FaithfulnessMetric(
    threshold=0.65,          # generous floor to avoid noise-driven failures
    model="gpt-4o-mini",
    include_reason=True,     # always log why
    strict_mode=False,       # allow graded scoring, not binary judge output
)

CI wiring: fail fast, fail specific

The last organizational piece is how the suite plugs into CI. Two patterns matter most.

First, run the cheapest, highest-signal checks first. Structural test cases that don't call an LLM at all — schema validation on retrieved chunks, checking that actual_output is non-empty, verifying tool-call arguments parse as valid JSON — should run before any metric that costs an API call and thirty seconds. Fail the build on those before spending money on LLM judges.

# .github/workflows/eval.yml (excerpt)
jobs:
  structural-checks:
    runs-on: ubuntu-latest
    steps:
      - run: pytest tests/eval -m "not smoke and not nightly and not expensive" -k structural

  smoke-evals:
    needs: structural-checks
    runs-on: ubuntu-latest
    steps:
      - run: deepeval test run tests/eval -m smoke

Second, make CI output map back to the folder structure you built in section one. A failing tests/eval/retrieval/test_context_recall.py in the GitHub Actions summary should be enough, on its own, for the retrieval owner to know they're needed — without anyone having to scroll through a wall of undifferentiated metric output to figure out which component actually regressed.

Keep the suite worth trusting

None of this is about ceremony for its own sake. A DeepEval suite earns trust the same way a unit test suite does: failures mean something specific, passes mean something specific, and the cost of adding a new case is proportional to the value of what it checks. Organize by component, keep one intent per test, centralize your metric configuration, separate data from logic, tier your CI runs, and budget for LLM judge noise deliberately instead of discovering it in production.

Do that, and the suite that started as one file with five test cases can grow to thousands of assertions across a dozen files without anyone dreading the day they have to open it. If you want a structured, hands-on walkthrough of DeepEval — from your first LLMTestCase through metric selection, custom metrics, and CI integration — check out the DeepEval Tutorial course, built to take you from a single test file to exactly the kind of suite described here.