DeepEval vs Promptfoo: Two Approaches to LLM Testing

The two-tool problem nobody warns you about

You ship a prompt change, the demo looks great, and three days later a user reports the chatbot confidently invented a refund policy that doesn't exist. This is the moment most teams discover that "it works on my machine" doesn't translate to LLM applications. Traditional unit tests check for exact equality. LLM outputs are probabilistic, verbose, and often correct in five different phrasings. You need a testing framework built for that reality, and two names keep coming up: DeepEval and Promptfoo.

Both are open-source, both let you define test cases and run them against your LLM pipeline, and both produce pass/fail reports you can wire into CI. But they come from different starting points. DeepEval grew out of the Python ML testing world and reads like pytest for LLMs. Promptfoo grew out of prompt engineering and red-teaming, and reads like a YAML-driven test matrix you can point at any provider. If you've only used one of them, you've probably assumed the other works the same way. It doesn't, and the differences matter more than the marketing pages suggest.

This article walks through both tools honestly: what they're good at, where they get awkward, and how to decide which one (or both) belongs in your evaluation stack.

What DeepEval actually is

DeepEval is a Python library, installed with pip install deepeval, that lets you write LLM evaluations as pytest-style test functions. If your team already writes Python tests for the rest of the application, DeepEval slots into that workflow with almost no context switch. A test case wraps an input, the actual output from your LLM, and optionally a retrieval context or expected output, and you score it against one or more metrics.

from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase

def test_support_bot_answer():
    test_case = LLMTestCase(
        input="How do I reset my password?",
        actual_output=call_support_bot("How do I reset my password?"),
        retrieval_context=[
            "To reset your password, go to Settings > Security > Reset Password."
        ],
    )

    relevancy = AnswerRelevancyMetric(threshold=0.7)
    faithfulness = FaithfulnessMetric(threshold=0.8)

    assert_test(test_case, [relevancy, faithfulness])

Run it with deepeval test run test_support_bot.py and you get a report showing which metrics passed, the numeric score, and a reasoning string explaining *why* the LLM judge scored it that way. That last part is the detail people underestimate: DeepEval's metrics are mostly LLM-as-judge under the hood (using GPT-4, Claude, or a local model you configure), and it surfaces the judge's reasoning rather than just a number. When a faithfulness check fails, you get a sentence explaining which claim in the output wasn't supported by the retrieval context, which saves a lot of guessing.

DeepEval ships a genuinely large metric library out of the box: answer relevancy, faithfulness, contextual precision/recall, hallucination, toxicity, bias, summarization quality, and G-Eval, which lets you define a custom rubric in plain English and have it compiled into a scoring chain. For RAG pipelines specifically, the contextual metrics (precision, recall, relevancy) are more granular than what most competing tools offer — they'll tell you separately whether your retriever pulled the right chunks and whether the generator actually used them.

What Promptfoo actually is

Promptfoo is a CLI-first tool, installed via npx promptfoo@latest init or npm install -g promptfoo, built around a YAML configuration file rather than test code. You declare providers (which models or endpoints to test), prompts (which can be templated), and test cases with assertions, and Promptfoo runs the full matrix — every prompt against every provider against every test case — and renders a side-by-side comparison in a local web UI.

# promptfooconfig.yaml
providers:
  - openai:gpt-4o-mini
  - anthropic:claude-3-5-sonnet-20241022

prompts:
  - "Answer the support question concisely: {{question}}"

tests:
  - vars:
      question: "How do I reset my password?"
    assert:
      - type: contains
        value: "Settings"
      - type: llm-rubric
        value: "The answer should mention navigating to account security settings"
      - type: cost
        threshold: 0.01

Run promptfoo eval and it executes the matrix, then promptfoo view opens a browser UI where you can literally see each model's output in a grid, color-coded pass/fail per cell. This is Promptfoo's signature move: it was built to answer "which model or which prompt variant performs best," not just "does this one pipeline pass or fail." If you're deciding between GPT-4o-mini and Claude Haiku for a cost-sensitive feature, or A/B testing two system prompt drafts, the grid view makes the comparison visual and immediate in a way no amount of scrolling through pytest output replicates.

Promptfoo's assertion library is broad and mixes deterministic checks (contains, equals, regex, javascript for custom logic, cost, latency) with model-graded ones (llm-rubric, similar using embeddings, factuality, answer-relevancy). It also has first-class support for red-teaming: promptfoo redteam init scaffolds adversarial test generation for jailbreaks, prompt injection, and PII leakage, which is a genuinely different use case from correctness testing and one DeepEval doesn't cover as a built-in workflow.

Setup and mental model

The fastest way to feel the difference is to notice what each tool assumes you already have. DeepEval assumes you have a Python function that calls your LLM application — your actual RAG pipeline, agent, or chat handler — and you're writing tests *around* that function, the same way you'd test any other piece of application logic. It has no concept of "providers" as a first-class abstraction; you call your own code inside the test case.

Promptfoo assumes the opposite: you often don't have application code yet, you have a prompt and a model choice you want to evaluate. Its provider abstraction (openai:, anthropic:, ollama:, or a custom HTTP/exec provider) means you can run an eval without writing a single line of application code. You can also point a provider at your own deployed API with an exec: or custom JavaScript provider if you do have a full pipeline, but that's an addition to the core model, not the default path.

This difference has a practical consequence: teams evaluating "which model should we use" or "which prompt wording performs better" tend to reach for Promptfoo first because the config-driven matrix is exactly the shape of that question. Teams evaluating "is our shipped RAG pipeline still passing quality gates" tend to reach for DeepEval first because it integrates as a normal test suite against real application code.

Metrics depth versus comparison breadth

Where DeepEval pulls ahead is metric sophistication for RAG and agent evaluation specifically. Its contextual precision and contextual recall metrics decompose retrieval quality into components most tools bundle into a single "relevance" score:

from deepeval.metrics import ContextualPrecisionMetric, ContextualRecallMetric

contextual_precision = ContextualPrecisionMetric(threshold=0.7)
contextual_recall = ContextualRecallMetric(threshold=0.7)

If your retriever returns five chunks and only two are actually relevant, contextual precision catches that even if the generator manages to write a decent answer anyway. DeepEval also has explicit support for evaluating multi-turn conversations and agent tool-use trajectories (checking whether an agent called the right tool with the right arguments in the right order), which is an area where Promptfoo's assertion model — built around single prompt-response pairs — is less natural to extend, though Promptfoo has been adding conversation-simulation support to close that gap.

Where Promptfoo pulls ahead is exactly the thing its name implies: comparing prompts and providers side by side, fast, without writing code. If your question is "does this prompt work better on Claude or GPT-4o," Promptfoo answers it as a first-class citizen. Doing the equivalent in DeepEval means writing a loop over providers yourself inside a test function — doable, but you're building the comparison harness rather than getting it for free.

CI integration

Both tools are designed to run in CI, and both work fine there, but the failure mode differs. DeepEval's deepeval test run command is a thin wrapper around pytest, so it inherits everything you already know about pytest exit codes, markers, and parallelization (-n auto via pytest-xdist works out of the box). A GitHub Actions step looks like:

- name: Run LLM evals
  run: |
    pip install -U deepeval
    deepeval test run tests/test_llm_quality.py

Promptfoo has its own CLI exit-code contract: promptfoo eval exits non-zero if any assertion fails, and you can set pass-rate thresholds so a single flaky model-graded check doesn't sink the whole build. It also ships a dedicated GitHub Action and a comment-on-PR integration that posts the eval results grid directly into the pull request, which is a nice touch if your reviewers aren't going to click into a CI log.

- name: Run promptfoo eval
  run: npx promptfoo@latest eval --no-cache
- name: Share eval results
  uses: promptfoo/promptfoo-action@v1
  with:
    prompts: prompts/*.txt
    config: promptfooconfig.yaml

Neither is harder to wire up than the other. The real CI consideration is cost and flakiness: both tools' model-graded assertions call an LLM judge, which means your CI run now depends on an external API, costs real money per run, and can vary between runs even on identical inputs if the judge model isn't pinned or temperature isn't controlled. Budget for this — teams that skip it are routinely surprised when a "flaky test" turns out to be a nondeterministic LLM judge rather than a bug in their code.

Where each one gets awkward

Being fair means naming the friction, not just the strengths.

DeepEval's awkward edges:

• The metric library is large, but tuning thresholds takes real iteration. A threshold=0.7 on FaithfulnessMetric isn't a universal constant — what counts as "faithful enough" varies by domain, and you'll spend time calibrating against a labeled dataset before you trust the gate.
• Because it's Python/pytest native, teams without a Python test suite already in place have to stand up that scaffolding first. It's not a heavy lift, but it's not zero either.
• Some of the more advanced features (synthetic dataset generation via Synthesizer, the hosted Confident AI dashboard) push toward a hosted product, and the line between "free open-source library" and "paid platform features" isn't always obvious from the docs on first read.

Promptfoo's awkward edges:

• The YAML config is great for small-to-medium test matrices, but it can get unwieldy once you have dozens of prompts, providers, and nested variable sets — you end up reaching for YAML anchors or splitting configs, which reintroduces complexity the format was supposed to avoid.
• Deep RAG-specific evaluation (checking retrieval quality independent of generation quality) isn't a first-class metric the way it is in DeepEval; you can approximate it with custom javascript assertions or llm-rubric prompts, but you're building what DeepEval gives you natively.
• The red-teaming module is powerful but is a genuinely different product surface from the eval/comparison workflow, and learning both halves of the tool well is more surface area than a team evaluating "just correctness testing" may want to take on initially.

A realistic combined workflow

In practice, the two tools aren't mutually exclusive, and treating this as a forced choice misses how teams actually use them. A pattern that works well: use Promptfoo early, during prompt and model selection, when you're iterating fast and want the visual grid to decide between three system prompt drafts and two candidate models. Once you've locked in a pipeline and shipped it, switch the ongoing quality gate to DeepEval, wired into your existing pytest suite and CI, so that faithfulness and answer relevancy regressions get caught the same way a broken API endpoint would.

# tests/test_regression_gate.py
from deepeval import assert_test
from deepeval.metrics import HallucinationMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
import pytest

test_cases = load_golden_dataset("golden_qa.json")  # your own loader

@pytest.mark.parametrize("case", test_cases)
def test_no_regression(case):
    actual = call_production_pipeline(case["input"])
    llm_test_case = LLMTestCase(
        input=case["input"],
        actual_output=actual,
        context=case["context"],
    )
    assert_test(llm_test_case, [
        HallucinationMetric(threshold=0.5),
        AnswerRelevancyMetric(threshold=0.7),
    ])

This split isn't a rule, it's a description of where each tool's design naturally pulls you. Some teams standardize on one tool for everything and it works fine — Promptfoo's assertion system is flexible enough to serve as a full regression gate if you don't need DeepEval's RAG-specific granularity, and DeepEval can be used for provider comparison if you're comfortable writing the loop yourself.

Which one should you actually pick

If your team is Python-first, already has pytest CI, and your core problem is "our RAG/agent pipeline needs a quality gate that catches hallucination and relevance regressions before merge," start with DeepEval. The metric depth for retrieval-augmented generation and the pytest-native workflow will feel like home immediately.

If your team needs to compare models or prompt variants quickly, wants a visual side-by-side without writing application code first, or needs red-teaming for jailbreak and injection testing as part of the same tool, start with Promptfoo. The YAML-driven matrix and the local web UI answer "which configuration wins" faster than any code-first approach will.

If you genuinely can't decide, that's a signal the answer is "both, at different stages of the same pipeline" rather than a sign you need to research longer. The cost of running both is low — they're both free, open source, and don't conflict with each other — and the cost of picking wrong and re-tooling later is a afternoon's work, not a rewrite.

The part neither tool solves for you

Whichever framework you pick, the hard part of LLM testing was never the framework — it's building a golden dataset that actually represents your production traffic, calibrating thresholds against real failures instead of guessing at 0.7, and deciding what "good enough" means for your specific domain before you write a single assertion. Both DeepEval and Promptfoo will faithfully execute whatever test cases you give them; neither will tell you that your test cases are testing the wrong thing. That judgment still has to come from you, and it's the part most teams skip until an incident forces the conversation.

If you want to go deeper on building that judgment — writing metrics that actually catch regressions, structuring a golden dataset, and wiring DeepEval into a real CI pipeline end to end — our DeepEval Tutorial course on teachyou.ai walks through it with a working RAG project from first test case to production gate.