DeepEval and Pytest Fixtures: Reusable Test Setup Patterns
Why Your DeepEval Tests Get Messy Without Fixtures
If you have written more than a handful of DeepEval test cases, you have probably noticed a pattern creeping into your codebase: the same GEval metric instantiated in five different test files, the same OpenAI model wrapper constructed over and over, and the same boilerplate for loading golden datasets copy-pasted across modules. It works, until it doesn't. Someone changes the threshold on your answer relevancy metric in one file and forgets the other four. A teammate adds a new evaluation model and now half your suite is testing against gpt-4o while the other half quietly still uses gpt-3.5-turbo. Nobody notices until a demo goes sideways.
This is exactly the problem pytest fixtures were built to solve, and DeepEval, since it is built directly on top of pytest, gets this for free. Fixtures let you define setup logic once, declare dependencies explicitly, and control the lifetime of expensive objects like LLM clients and metric instances. Instead of scattering AnswerRelevancyMetric(threshold=0.7, model="gpt-4o") across your repository, you define it once in a fixture and every test that needs it simply asks for it by name.
In this article we will walk through how to build a layered fixture architecture for DeepEval test suites: metric fixtures, model fixtures, test case factories, conftest.py organization, and fixture scoping for performance. By the end you will have a pattern you can drop into any LLM evaluation project, whether you are testing a RAG pipeline, an agent, or a simple prompt-in-answer-out system.
A Quick Refresher on How DeepEval Uses Pytest
DeepEval does not reinvent test discovery or assertions. It hooks into pytest's existing machinery and adds two things: an assert_test function that runs your metrics against an LLMTestCase, and a custom test runner (deepeval test run) that adds parallelization, retries, and reporting on top of standard pytest.
A minimal DeepEval test looks like this:
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
def test_answer_relevancy():
metric = AnswerRelevancyMetric(threshold=0.7)
test_case = LLMTestCase(
input="What is the capital of France?",
actual_output="The capital of France is Paris.",
retrieval_context=["Paris is the capital and most populous city of France."]
)
assert_test(test_case, [metric])This is fine for a single test. But every real DeepEval suite needs dozens, sometimes hundreds, of these, covering different intents, edge cases, and regression scenarios. That's where fixtures come in, because pytest fixtures are just functions decorated with @pytest.fixture that pytest injects into any test function requesting them by parameter name. DeepEval tests are regular pytest test functions, so every fixture pattern you already know applies directly.
Building Your First Metric Fixture
The most common source of duplication in DeepEval suites is metric instantiation. Metrics like AnswerRelevancyMetric, FaithfulnessMetric, and GEval all take configuration (threshold, model, strict mode) that should be consistent across your test run. Move that configuration into a fixture.
# conftest.py
import pytest
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
@pytest.fixture
def answer_relevancy_metric():
return AnswerRelevancyMetric(
threshold=0.7,
model="gpt-4o-mini",
include_reason=True
)
@pytest.fixture
def faithfulness_metric():
return FaithfulnessMetric(
threshold=0.8,
model="gpt-4o-mini",
include_reason=True
)Now any test file in the same directory (or subdirectory) can request these fixtures by name, without an import:
# test_rag_answers.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
def test_paris_capital(answer_relevancy_metric, faithfulness_metric):
test_case = LLMTestCase(
input="What is the capital of France?",
actual_output="The capital of France is Paris.",
retrieval_context=["Paris is the capital and most populous city of France."]
)
assert_test(test_case, [answer_relevancy_metric, faithfulness_metric])Notice what changed here. The test function no longer knows or cares what threshold the relevancy metric uses, or which model is judging faithfulness. That configuration lives in one place. When your evaluation model gets deprecated (this happens more often than you'd think with hosted LLM providers), you update conftest.py once and every test in the suite is instantly using the new model.
Parameterizing Fixtures for Multiple Configurations
Sometimes you genuinely need different metric configurations in different parts of your suite; maybe your production-critical checkout flow needs a stricter threshold than your general FAQ chatbot. Pytest fixtures support parameterization through request.param, and you can also just define multiple named fixtures for different strictness levels.
# conftest.py
import pytest
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams
@pytest.fixture
def correctness_metric_strict():
return GEval(
name="Correctness",
criteria="Determine whether the actual output is factually correct given the expected output.",
evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
threshold=0.9,
model="gpt-4o"
)
@pytest.fixture
def correctness_metric_lenient():
return GEval(
name="Correctness",
criteria="Determine whether the actual output is factually correct given the expected output.",
evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
threshold=0.6,
model="gpt-4o-mini"
)An alternative, more scalable approach is to build a factory fixture: a fixture that returns a function, which you then call inside the test with whatever parameters you need at that moment. This is one of the most underused pytest patterns in DeepEval suites.
# conftest.py
import pytest
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams
@pytest.fixture
def make_correctness_metric():
def _make(threshold=0.7, model="gpt-4o-mini"):
return GEval(
name="Correctness",
criteria="Determine whether the actual output is factually correct given the expected output.",
evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
threshold=threshold,
model=model
)
return _make# test_correctness.py
def test_high_stakes_answer(make_correctness_metric):
metric = make_correctness_metric(threshold=0.95, model="gpt-4o")
# ... build test case and assertThe factory pattern gives you the reusability of a fixture with the flexibility of a constructor call, and it keeps one-off overrides from forcing you to define a whole new fixture every time a test needs slightly different behavior.
Test Case Fixtures and Data Factories
Metrics are only half the story. The other half is building LLMTestCase objects, and in real projects these often come from a golden dataset, a CSV, a JSON file, or a database of production traces you've sampled for regression testing. Loading that data inside every test function is wasteful and brittle. Instead, load it once with a fixture and let pytest manage its lifecycle.
# conftest.py
import json
import pytest
from deepeval.test_case import LLMTestCase
@pytest.fixture(scope="session")
def golden_dataset():
with open("tests/data/golden_qa.json") as f:
return json.load(f)
@pytest.fixture
def make_test_case():
def _make(input_text, actual_output, expected_output=None, retrieval_context=None):
return LLMTestCase(
input=input_text,
actual_output=actual_output,
expected_output=expected_output,
retrieval_context=retrieval_context or []
)
return _makeYou can then combine the session-scoped dataset with the factory fixture and pytest's own parametrization to generate many test cases from one function:
# test_golden_dataset.py
import pytest
from deepeval import assert_test
def test_golden_qa_pairs(golden_dataset, make_test_case, answer_relevancy_metric):
for record in golden_dataset:
test_case = make_test_case(
input_text=record["input"],
actual_output=record["actual_output"],
expected_output=record.get("expected_output"),
retrieval_context=record.get("retrieval_context")
)
assert_test(test_case, [answer_relevancy_metric])If you want each golden record reported as its own pytest test (which gives you much better failure visibility in CI), use pytest.mark.parametrize fed by a fixture-adjacent helper function instead of looping inside a single test body:
# test_golden_dataset_parametrized.py
import json
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
def load_golden_records():
with open("tests/data/golden_qa.json") as f:
return json.load(f)
@pytest.mark.parametrize("record", load_golden_records())
def test_golden_record(record, answer_relevancy_metric):
test_case = LLMTestCase(
input=record["input"],
actual_output=record["actual_output"],
retrieval_context=record.get("retrieval_context", [])
)
assert_test(test_case, [answer_relevancy_metric])This gives you one pytest test ID per golden record in your report output, which makes it trivial to spot exactly which inputs regressed after a prompt change.
Fixture Scoping: Function, Class, Module, and Session
Fixture scope is where a lot of DeepEval suites either waste money or introduce subtle bugs. By default, pytest fixtures are function-scoped, meaning they are recreated for every single test function. For a lightweight object like an LLMTestCase, that's fine. For something like an LLM client wrapper that needs to authenticate or a locally hosted evaluation model that takes seconds to load into memory, recreating it per test is a real cost, both in wall-clock time and, if you're hitting a hosted API for setup calls, in actual dollars.
Here's how the four scopes map onto typical DeepEval fixture needs:
- function (default): use for
LLMTestCaseobjects and anything that must be fresh per test to avoid state leaking between assertions. - class: use when you group related tests in a test class and want shared setup across just that group, for example all tests validating one specific chatbot intent.
- module: use for metrics that are expensive to construct but safe to reuse across every test in a file, such as a
GEvalmetric with a complex custom criteria string. - session: use for genuinely global, expensive resources: a locally-run evaluation model, a loaded golden dataset, or a shared API client configured once for the entire test run.
# conftest.py
import pytest
from deepeval.models import GPTModel
from deepeval.metrics import AnswerRelevancyMetric
@pytest.fixture(scope="session")
def eval_model():
# Constructed once for the entire test run.
return GPTModel(model="gpt-4o-mini")
@pytest.fixture(scope="module")
def relevancy_metric(eval_model):
return AnswerRelevancyMetric(threshold=0.7, model=eval_model)Notice the dependency chain: relevancy_metric depends on eval_model, and pytest resolves this automatically. The session-scoped model is built exactly once, then reused by every module-scoped metric fixture that asks for it, which is reused by every test function in that module. This is the single biggest lever for cutting down DeepEval suite runtime when your evaluation model has any kind of cold-start cost.
One caution here: if your metric objects hold mutable state across calls (some custom metrics track intermediate reasoning), be careful sharing them at broader scope than function level, since one test's evaluation could bleed into another's. Read your custom metric's measure implementation before you widen its fixture scope.
Organizing conftest.py Across a Larger Test Suite
Once a DeepEval suite grows past a handful of files, a single flat conftest.py starts to feel unwieldy. Pytest supports nested conftest.py files, and fixtures defined in a parent directory are automatically visible to tests in child directories, without any import statement. This lets you build a layered fixture architecture that mirrors your test organization.
A typical layout for a mid-sized DeepEval project:
tests/
conftest.py # global: eval_model, base metrics, make_test_case factory
rag/
conftest.py # RAG-specific: retrieval_context fixtures, faithfulness metric
test_retrieval_quality.py
test_answer_grounding.py
chatbot/
conftest.py # chatbot-specific: conversation history fixtures, tone metrics
test_intent_handling.py
test_tone_consistency.py
agents/
conftest.py # agent-specific: tool-call metrics, task completion fixtures
test_tool_selection.py# tests/conftest.py
import pytest
from deepeval.models import GPTModel
@pytest.fixture(scope="session")
def eval_model():
return GPTModel(model="gpt-4o-mini")# tests/rag/conftest.py
import pytest
from deepeval.metrics import FaithfulnessMetric, ContextualPrecisionMetric
@pytest.fixture(scope="module")
def faithfulness_metric(eval_model):
return FaithfulnessMetric(threshold=0.8, model=eval_model)
@pytest.fixture(scope="module")
def contextual_precision_metric(eval_model):
return ContextualPrecisionMetric(threshold=0.7, model=eval_model)Here, tests/rag/conftest.py uses eval_model without importing it, because pytest walks up the directory tree looking for fixtures by name. This keeps global, expensive setup at the top level and domain-specific metric configuration close to the tests that actually use it. When a new engineer opens tests/agents/conftest.py, they see exactly the fixtures relevant to agent testing, not a wall of unrelated RAG fixtures.
Handling Setup, Teardown, and Environment Variables
DeepEval tests often need environment configuration, most commonly API keys for whichever LLM provider you're using as a judge. Fixtures are also the right place to validate that required environment variables exist before your suite burns through API calls only to fail on the first assertion.
# conftest.py
import os
import pytest
@pytest.fixture(scope="session", autouse=True)
def validate_environment():
required_vars = ["OPENAI_API_KEY"]
missing = [v for v in required_vars if not os.environ.get(v)]
if missing:
pytest.exit(f"Missing required environment variables: {missing}", returncode=1)The autouse=True flag means every test session runs this check automatically, without any test function needing to request it explicitly. This is a good place to fail fast, rather than discovering thirty tests in that your API key was never set.
Fixtures also support teardown logic using yield instead of return. This matters if you're logging evaluation results to a file, cleaning up temporary vector store collections, or closing a database connection used to fetch test data.
# conftest.py
import pytest
import json
from datetime import datetime
@pytest.fixture(scope="session")
def results_logger():
results = []
yield results
# Teardown runs after the whole session finishes.
with open(f"eval_results_{datetime.now().isoformat()}.json", "w") as f:
json.dump(results, f, indent=2)# test_logged_eval.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
def test_and_log(answer_relevancy_metric, results_logger):
test_case = LLMTestCase(
input="What is the capital of France?",
actual_output="The capital of France is Paris.",
retrieval_context=["Paris is the capital and most populous city of France."]
)
answer_relevancy_metric.measure(test_case)
results_logger.append({
"input": test_case.input,
"score": answer_relevancy_metric.score,
"reason": answer_relevancy_metric.reason
})
assert answer_relevancy_metric.score >= answer_relevancy_metric.thresholdEverything before the yield statement runs as setup; everything after runs as teardown once the fixture's scope ends. For a session-scoped fixture like this one, that means the results file gets written exactly once, after every test in the run has appended its outcome.
Common Fixture Pitfalls in DeepEval Suites
A few mistakes show up repeatedly in DeepEval test suites, and they're worth calling out directly.
- Over-scoping mutable metrics. Sharing a metric instance at session scope is fine if the metric is stateless between calls, but some custom
GEvalconfigurations accumulate reasoning traces or verbose logs internally. Test this before widening scope. - Hiding thresholds too deep. If a fixture nests three levels deep before you find the actual threshold value, debugging a failing test becomes a scavenger hunt. Keep the most commonly tuned values (thresholds, models) near the top of your fixture chain, or expose them as factory parameters.
- Forgetting fixture finalization order. Pytest tears down fixtures in reverse order of setup. If fixture B depends on fixture A, A is torn down after B. This matters when a teardown step in one fixture assumes another fixture's resource is still alive.
- Not using `autouse` for validation, but overusing it for setup.
autouse=Trueis great for environment checks, but making every metric fixtureautousemeans every test constructs metrics it doesn't need, slowing down the suite for no benefit. - Re-fetching remote data per test. If your golden dataset lives in a database or an external file store, fetch it once with a session-scoped fixture rather than re-fetching in every test function.
Avoiding these five issues alone will resolve the majority of "why is my DeepEval suite slow" or "why did this test fail only in CI" questions that come up in practice.
Putting It All Together
The pattern that tends to work best across real DeepEval projects looks like this: a root conftest.py with session-scoped, expensive resources (evaluation models, environment validation, shared datasets); domain-specific conftest.py files nested under each test category with module-scoped metric fixtures tuned for that domain; and factory fixtures wherever a test needs to construct a metric or test case with parameters that vary per call. Function-scoped LLMTestCase objects stay fresh per test by default, which is exactly the safety you want since test cases should never leak state between assertions.
None of this requires anything beyond standard pytest, which is precisely why it fits so naturally with DeepEval: you are not learning a new testing framework, you are applying fixture design you may already know from testing regular Python applications to the specific problem of evaluating LLM outputs. Once the fixture layer is in place, adding a new evaluation test is often just a few lines: request the metrics you need, build or fetch a test case, call assert_test.
If you want a guided, hands-on walkthrough of building a full DeepEval suite from scratch, including fixture architecture, custom metrics, CI integration, and RAG-specific evaluation patterns, check out the DeepEval Tutorial course on teachyou.ai. It covers exactly the patterns in this article with real project code you can adapt directly into your own test suite.
BootcampA 30-day guided bootcamp: build, harden and ship a production autonomous agent from scratch.
AI AgentsUnderstand how AI agents really work: the loop, the tools, the memory, and why most agent projects fail.
Related reading