DeepEval Tutorial: Unit Testing Your LLM App Like Pytest
You shipped a RAG chatbot last sprint. It worked in your demo. Then a teammate changed the system prompt "just slightly," a vector store got re-indexed, and now support tickets are rolling in about hallucinated refund policies. Nobody caught it because nobody had a test suite. This is the normal life cycle of an LLM feature without evals, and it is exactly the gap DeepEval was built to close. If you already know pytest, you already know 80% of DeepEval — the rest is a handful of metrics designed specifically for judging natural language output instead of exact-match assertions.
This tutorial walks through installing DeepEval, writing your first test case, running a built-in metric against real output, setting a pass/fail threshold, reading a failed report, and wiring the whole thing into CI so regressions get caught before they reach production.
Why "unit testing" even applies to LLMs
Traditional unit tests work because outputs are deterministic: add(2, 2) always returns 4. LLM outputs are not deterministic, and "correctness" is fuzzy — there are many valid phrasings of a correct answer and many plausible-sounding wrong ones. That fuzziness is exactly why teams skip testing LLM apps altogether and just eyeball outputs in a notebook.
DeepEval closes that gap by replacing exact-match assertions with metric-based assertions. Instead of asserting output == expected_output, you assert something like "the output is at least 80% relevant to the input" or "the output does not contradict the retrieved context." Under the hood, most of these metrics use an LLM-as-a-Judge — a second LLM call that scores your app's output against a rubric. You still get a boolean pass/fail, a real number score, and a reason string, all wrapped in familiar pytest syntax: assert, fixtures, parametrize, markers, the works.
If your team already runs pytest in CI, DeepEval slots into that pipeline with almost no new infrastructure. That's the whole pitch, and it's a good one.
It also reframes how your team talks about quality. "The bot feels worse since the update" is not actionable. "Faithfulness on the refund-policy golden set dropped from 0.94 to 0.71 after the last prompt change" is a bug report an engineer can act on the same afternoon. That shift — from vibes to numbers you can diff between commits — is the real reason to adopt this workflow, not the tooling itself.
Installing DeepEval
Start with a clean virtual environment so metric dependencies don't collide with your app's dependencies.
python -m venv .venv
source .venv/bin/activate
pip install deepeval
# Optional: log in to sync results to Confident AI's dashboard
deepeval loginDeepEval defaults to using OpenAI models as the judge for its metrics, so you'll need an API key available as an environment variable:
export OPENAI_API_KEY="sk-..."You can swap in a different judge model (Anthropic, a local model via Ollama, or your own wrapper class) later — more on that in the CI section, since you don't want every pull request burning judge-model tokens against a frontier model by default.
Verify the install:
deepeval --versionIf that prints a version number, you're ready to write your first test.
One thing that trips people up coming from regular pytest: because most metrics call out to a real LLM API, your test suite now has network dependencies and per-run cost, even for a "unit" test. That's a genuine trade-off. It means you'll want to think about test scope more deliberately than you do for pure Python logic — more on that in the CI section — but it does not mean the tests are any less real. A failing faithfulness test is telling you something just as concrete as a failing assertion on a parser function; it just costs a fraction of a cent to find out.
Anatomy of a test case
DeepEval's core primitive is LLMTestCase. It's a plain data container that holds everything a metric needs to score one interaction with your app:
- input — what the user asked
- actual_output — what your LLM app actually returned
- expected_output — what a correct answer looks like (used by some metrics, optional for others)
- retrieval_context — the chunks your RAG pipeline retrieved, if applicable
- context — ground-truth context, when you have it independent of retrieval
Here's the smallest possible example, run outside of pytest just to see the shape of things:
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
from deepeval import evaluate
test_case = LLMTestCase(
input="What is the refund window for annual subscriptions?",
actual_output=(
"Annual subscriptions can be refunded within 30 days "
"of purchase, minus any usage-based fees."
),
retrieval_context=[
"Our refund policy: annual plans are refundable within "
"30 days of the original purchase date.",
"Monthly plans are non-refundable after the billing cycle starts.",
],
)
relevancy_metric = AnswerRelevancyMetric(threshold=0.8)
evaluate(test_cases=[test_case], metrics=[relevancy_metric])Running this prints a score (a float between 0 and 1), a pass/fail boolean against your threshold, and — crucially — a reason string explaining why the judge scored it that way. That reason field is what makes debugging a failed eval tractable instead of a black box.
Notice what retrieval_context is doing here. It's not just decoration — it's the ground truth the FaithfulnessMetric and contextual metrics will compare actual_output against later. If you skip it, you can still run AnswerRelevancyMetric (which only needs input and actual_output), but you lose the ability to catch hallucinations, because there's nothing to check the claim against. Get in the habit of capturing whatever your retriever actually returned, not a hand-written approximation of it — the whole value of the faithfulness check depends on it being the real context your app saw at generation time.
Writing your first pytest-style test
The whole point of DeepEval is that this doesn't have to live in a throwaway script. It lives in test_*.py files, next to your regular test suite, and runs with the same pytest command your team already uses.
# test_support_bot.py
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from app.chatbot import answer_question # your actual app code
def test_refund_policy_answer_is_relevant():
user_input = "What is the refund window for annual subscriptions?"
actual_output = answer_question(user_input)
test_case = LLMTestCase(
input=user_input,
actual_output=actual_output,
retrieval_context=[
"Annual plans are refundable within 30 days of purchase.",
],
)
relevancy = AnswerRelevancyMetric(threshold=0.8)
assert_test(test_case, [relevancy])
def test_refund_policy_answer_is_faithful_to_context():
user_input = "What is the refund window for annual subscriptions?"
actual_output = answer_question(user_input)
test_case = LLMTestCase(
input=user_input,
actual_output=actual_output,
retrieval_context=[
"Annual plans are refundable within 30 days of purchase.",
],
)
faithfulness = FaithfulnessMetric(threshold=0.9)
assert_test(test_case, [faithfulness])Two things worth calling out here. First, answer_question() is your real application function — DeepEval doesn't care whether that's a call to OpenAI, a LangChain chain, a LlamaIndex query engine, or a homegrown RAG pipeline. It just needs a string in, string out. Second, assert_test behaves exactly like a normal assert: if the metric fails its threshold, the test fails, pytest reports it in red, and your CI job exits non-zero. No custom test runner, no bespoke reporting format to learn.
Run it the same way you'd run anything else:
pytest test_support_bot.py -vReusing fixtures and markers like normal pytest
Because DeepEval tests are just pytest tests, every organizational tool you already rely on works unchanged. That includes conftest.py fixtures for shared setup, pytest.mark for slicing the suite, and -k for running a subset by name.
A common pattern is to put your app client and default metrics behind fixtures so individual tests stay short:
# conftest.py
import pytest
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from app.chatbot import SupportBot
@pytest.fixture(scope="session")
def bot():
return SupportBot(model="gpt-4o-mini")
@pytest.fixture
def relevancy_metric():
return AnswerRelevancyMetric(threshold=0.8, include_reason=True)
@pytest.fixture
def faithfulness_metric():
return FaithfulnessMetric(threshold=0.9, include_reason=True)# test_support_bot.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
@pytest.mark.eval
@pytest.mark.rag
def test_refund_policy(bot, faithfulness_metric):
user_input = "What is the refund window for annual subscriptions?"
actual_output, retrieval_context = bot.answer_with_context(user_input)
test_case = LLMTestCase(
input=user_input,
actual_output=actual_output,
retrieval_context=retrieval_context,
)
assert_test(test_case, [faithfulness_metric])Register the eval and rag markers in pytest.ini or pyproject.toml the same way you would for any other custom marker, and now you can run just the eval suite, or just the RAG-specific subset, without touching your other tests:
pytest -m eval -v
pytest -m "eval and rag" -vThis matters more than it looks. As your golden set grows past a hundred cases, you'll want fast local iteration on a handful of cases (-k test_refund_policy) while CI runs the full marked suite. Treating your eval tests as first-class pytest citizens — not a separate bolt-on script — is what keeps that workflow cheap.
Picking the right built-in metric
DeepEval ships a library of metrics so you're not writing "is this text good" logic from scratch. The ones you'll reach for constantly:
- AnswerRelevancyMetric — does the output actually address the input, without padding or going off-topic
- FaithfulnessMetric — does the output avoid contradicting or fabricating beyond the retrieval context (your hallucination detector)
- ContextualPrecisionMetric and ContextualRecallMetric — is your retriever pulling the right chunks, ranked well, for RAG pipelines
- HallucinationMetric — a more direct check against a known-good context for factual grounding
- GEval — a custom rubric metric where you describe a criteria in plain English and DeepEval builds an LLM judge around it
That last one, GEval, is worth a dedicated example because it's the escape hatch for anything domain-specific that a built-in metric doesn't cover well — like "does this response match our brand's tone" or "does this legal summary avoid giving definitive legal advice."
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
tone_metric = GEval(
name="Supportive Tone",
criteria=(
"Determine whether the actual output responds to the customer "
"in a calm, empathetic tone, and never blames the customer "
"for their confusion."
),
evaluation_params=[
LLMTestCaseParams.INPUT,
LLMTestCaseParams.ACTUAL_OUTPUT,
],
threshold=0.7,
)
test_case = LLMTestCase(
input="Your product charged me twice, this is ridiculous.",
actual_output=(
"I'm sorry about the duplicate charge — that's frustrating, "
"and I've already flagged it for a refund. You should see it "
"reversed within 3-5 business days."
),
)
tone_metric.measure(test_case)
print(tone_metric.score, tone_metric.reason)Start with AnswerRelevancyMetric and FaithfulnessMetric for any RAG or Q&A system — they cover the two failure modes that hurt the most (irrelevant answers and confident hallucination). Add GEval metrics once you know the specific ways your app tends to go wrong, rather than trying to cover everything on day one.
A subtlety worth internalizing early: metrics like AnswerRelevancyMetric and FaithfulnessMetric don't just return a single opaque number under the hood. Conceptually, they decompose the output into individual claims or statements, check each one against the input or context, and roll the per-claim results up into a score. That decomposition is exactly why the reason string can point at a specific sentence instead of just saying "looks off." It's also why these metrics cost more than a single LLM call — budget for that when you're estimating CI spend.
Setting thresholds that mean something
The threshold parameter is where a lot of teams get lazy and just copy 0.7 from a tutorial without thinking about it. Don't do that. A threshold is a product decision, not a technical one.
A few practical guidelines:
- Start permissive, then tighten. Run your metric against 20-30 real examples first and look at the score distribution before you lock in a number. If your genuinely good answers are scoring 0.75-0.85, a threshold of 0.9 will fail things that are actually fine.
- Faithfulness deserves a higher bar than relevancy. A slightly-off-topic answer is annoying; a hallucinated refund policy is a support ticket and possibly a chargeback. We typically set
FaithfulnessMetric(threshold=0.9)or higher and are more lenient with relevancy at0.7-0.8. - Different thresholds for different environments. A stricter threshold on the
mainbranch than on a feature branch is a completely reasonable pattern — treat it like code coverage gates.
faithfulness = FaithfulnessMetric(
threshold=0.9,
include_reason=True, # always keep this on while tuning
)Keep include_reason=True on every metric while you're still calibrating thresholds. The reason string is the difference between "this failed" and "this failed because the output claimed a 60-day window when the context said 30 days" — the second one tells you whether to fix your prompt, your retriever, or your threshold.
It also helps to version your thresholds the same way you version anything else that affects behavior — in a config file, reviewed in pull requests, not hardcoded inline across a dozen test files. A small eval_config.py with named threshold profiles ("strict", "default", "lenient") makes it trivial to bump every faithfulness check at once when you upgrade your judge model or change your retriever, instead of hunting through test files for magic numbers.
# eval_config.py
THRESHOLDS = {
"faithfulness": 0.90,
"answer_relevancy": 0.80,
"contextual_precision": 0.75,
}Import from this single source of truth in every test file, and a threshold change becomes a one-line diff instead of a search-and-replace across the repo.
Running the suite and reading a failure
Run the full suite the way you'd run any pytest job:
pytest tests/ -v --tb=shortA passing faithfulness test looks unremarkable — green dot, move on. A failing one is where DeepEval earns its keep. A typical failure in verbose output looks like this:
FAILED tests/test_support_bot.py::test_refund_policy_answer_is_faithful_to_context
Metric: Faithfulness (GPT-4 as judge)
Score: 0.62 (threshold: 0.9)
Reason: The actual output states a "60-day" refund window, but the
retrieval context only supports a "30-day" window. This claim is not
grounded in the provided context and is likely a hallucination.
Input: What is the refund window for annual subscriptions?
Actual Output: Annual subscriptions can be refunded within 60 days...
Retrieval Context: ['Annual plans are refundable within 30 days of purchase.']That's a genuinely actionable bug report. You now know exactly which fact was fabricated, which document should have grounded it, and roughly how far off the model was. Compare that to the alternative — a support ticket three weeks later saying "the bot told me the wrong policy" with no reproduction steps.
When a test fails, resist the urge to just raise the threshold until it passes. First ask: is this a real regression (bad retrieval, bad prompt, model drift), or is the metric itself miscalibrated for this case? DeepEval's reason output usually makes that distinction obvious within a few seconds of reading it.
It's worth building a habit of triaging failures into three buckets before you touch anything: a retrieval bug (the wrong chunk was fetched, so no amount of prompt tweaking will fix it), a generation bug (the right context was retrieved but the model still drifted from it), or a metric miscalibration (the output is actually fine and the threshold or judge prompt needs adjusting). Conflating these three is the single most common way teams waste a week chasing a "hallucination" that was actually a retrieval index that never got rebuilt after a content update.
Parametrizing across a whole eval dataset
Real evaluation suites aren't one test case — they're dozens or hundreds, covering edge cases, adversarial inputs, and known-tricky queries. Use pytest's parametrize exactly like you would for any other data-driven test:
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric
from app.chatbot import answer_question
GOLDEN_SET = [
{
"input": "What is the refund window for annual subscriptions?",
"context": ["Annual plans are refundable within 30 days of purchase."],
},
{
"input": "Can I get a refund on a monthly plan?",
"context": ["Monthly plans are non-refundable after billing starts."],
},
{
"input": "Do you offer refunds for enterprise contracts?",
"context": ["Enterprise refunds are handled case-by-case by account managers."],
},
]
@pytest.mark.parametrize("case", GOLDEN_SET)
def test_faithfulness_across_golden_set(case):
actual_output = answer_question(case["input"])
test_case = LLMTestCase(
input=case["input"],
actual_output=actual_output,
retrieval_context=case["context"],
)
faithfulness = FaithfulnessMetric(threshold=0.85)
assert_test(test_case, [faithfulness])This is the pattern that actually scales. Every time support flags a bad response, add it to GOLDEN_SET as a regression test. Within a few months you have a living dataset that encodes every mistake your bot has ever made in production, and pytest runs the whole thing on every commit.
For anything beyond a few dozen cases, move GOLDEN_SET out of the test file and into a dataset.jsonl or a Dataset object DeepEval can load directly, and pull it from wherever your team already reviews content changes — a shared spreadsheet exported to JSON, a Notion database, or a dedicated eval-data repo. The exact storage format matters less than the discipline: golden sets rot the moment nobody owns adding to them, so put a name on that responsibility the same way you would for maintaining test fixtures in any other codebase.
Integrating DeepEval into CI
This is the payoff — none of the above matters if it only runs on your laptop. Here's a minimal GitHub Actions workflow that runs the eval suite on every pull request:
# .github/workflows/deepeval.yml
name: LLM Evals
on:
pull_request:
branches: [main]
jobs:
deepeval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install dependencies
run: |
pip install -r requirements.txt
pip install deepeval
- name: Run LLM eval suite
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
pytest tests/evals/ -v --tb=shortA few things that matter in practice once you're running this on every PR instead of on your laptop:
- Cost control. Every metric call is at least one LLM call, sometimes more (some metrics decompose the output into claims and score each one). Keep your CI golden set focused — tens of cases, not thousands — and run the expensive, exhaustive sweep nightly on a schedule instead of on every push.
- Judge model choice. Using GPT-4-class judges on every PR adds up fast. Many teams use a cheaper, faster judge model for PR gating and reserve the strongest judge for a nightly or pre-release run.
- Flakiness. LLM-as-a-Judge scoring has some run-to-run variance. If a test is borderline-flaky, that's a signal your threshold is too tight for that case, not that DeepEval is broken — widen the threshold slightly or average over a couple of runs.
- Fail the build, don't just log it. The entire value of this pattern is that a regression blocks a merge the same way a broken unit test would. If your eval suite only produces a dashboard nobody checks, you've built a report, not a test suite.
Beyond the basics
Once the core loop is working — write a test case, pick a metric, set a threshold, run in CI — a few directions are worth exploring next: synthetic dataset generation for scaling up your golden set without hand-writing every case, conversational test cases for multi-turn agents instead of single-shot Q&A, and red-teaming metrics for adversarial robustness (prompt injection, jailbreak attempts). DeepEval supports all three, but they build on exactly the pattern you just learned — you don't need new mental models, just new metrics and test case shapes.
The underlying idea that makes all of this possible — using one LLM to grade another LLM's output on a rubric — is deceptively easy to get wrong. Judge models have their own biases, prompt sensitivity, and blind spots, and a badly designed judge prompt will confidently give you garbage scores that look precise. If you want to actually understand how these judges work, how to design rubrics that hold up under scrutiny, and how to validate that your judge agrees with human raters, that's the exact ground we cover in our "LLM-as-a-Judge" course at teachyou.ai — it's the natural next step after you've got DeepEval running green in CI.
BootcampA 30-day guided bootcamp: build, harden and ship a production autonomous agent from scratch.
AI AgentsUnderstand how AI agents really work: the loop, the tools, the memory, and why most agent projects fail.
Related reading