Common DeepEval Errors and How to Fix Them

Why DeepEval breaks more often than you'd expect

If you've wired DeepEval into a CI pipeline to evaluate your LLM app, you've probably already hit a wall that had nothing to do with your prompts or your model. It was an API key DeepEval couldn't find, a metric that silently returned 0.0 for every test case, or a pytest run that hung for two minutes before dying with a timeout you'd never seen before. DeepEval sits at the intersection of three moving parts — your test framework, an LLM-as-judge call, and (often) a hosted platform for storing results — and each of those parts fails in its own way.

None of this means DeepEval is fragile. It means the failure modes are learnable. Once you've seen the actual error text for the ten or so things that go wrong most often, you stop guessing and start fixing in under a minute. That's what this article is for: real error strings, why they happen, and the fix that actually works, not just "check your config" hand-waving.

We'll cover API key and authentication errors, metric-level failures like G-Eval returning nonsense scores, async and event-loop errors that show up when DeepEval runs inside notebooks or FastAPI apps, schema/JSON parsing errors from the judge model, dataset and golden-file mismatches, and the CI-specific issues that only appear once you move off your laptop. By the end, you'll have a mental checklist to run through the next time a DeepEval test suite goes red for a reason that has nothing to do with your actual application quality.

"Error: The API key provided is invalid" and other auth failures

This is the single most common first error, and it usually happens on someone's very first deepeval test run. The traceback looks something like this:

openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-proj-***. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

DeepEval's default metrics (AnswerRelevancyMetric, FaithfulnessMetric, GEval, and most of the others) use an LLM as a judge. Unless you've explicitly configured a different model, that judge is GPT-4o via the OpenAI API, and DeepEval looks for OPENAI_API_KEY in your environment. Three things usually go wrong:

• The key is set in a .env file, but nothing is loading that file before deepeval test run executes, so the environment variable is simply empty.
• The key is exported in one shell session but the CI runner uses a fresh, non-interactive shell that never sourced .zshrc/.bashrc.
• The key was set with deepeval set-openai-key, which writes to a local .deepeval config cache, but the working directory changed and DeepEval can't find that cache anymore.

The fix that works in virtually every case is to stop relying on ambient shell state and instead load env vars explicitly at the top of your test file:

import os
from dotenv import load_dotenv

load_dotenv()  # loads .env from cwd, no more "it works on my machine"

assert os.getenv("OPENAI_API_KEY"), "OPENAI_API_KEY is not set — check your .env file"

In CI (GitHub Actions, GitLab CI, etc.), set the key as a repository secret and inject it as an environment variable in the workflow step that runs deepeval test run, rather than assuming a .env file exists on the runner — it won't, because .env should be gitignored.

If you're using a custom judge model (Azure OpenAI, Anthropic, a local Ollama model), the same class of error shows up with a different vendor's exception type. The fix is identical in spirit: confirm the credential the custom model wrapper expects is actually present in the process environment DeepEval is running in, not just in your terminal history.

Metrics returning 0.0 or throwing a `TypeError` on `measure()`

The second-most-common issue is a metric that runs without crashing but gives you a score of exactly 0.0 for every single test case, even ones that are obviously good. This usually means the judge model's response couldn't be parsed into the structured format DeepEval expects, and DeepEval silently fell back to a zero score instead of surfacing the parsing failure loudly (in older versions especially).

The other variant is a hard crash:

TypeError: measure() missing 1 required positional argument: 'test_case'

This one is almost always a version mismatch between how you're calling the metric and the DeepEval version installed. In DeepEval 0.21+ the calling convention for metric.measure() changed slightly, and code copy-pasted from an older tutorial or blog post breaks. The fix is to pin your DeepEval version and match it to the API you're actually using:

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="What is the return policy?",
    actual_output="You can return any item within 30 days of purchase.",
    retrieval_context=["Our return policy allows returns within 30 days."],
)

metric = AnswerRelevancyMetric(threshold=0.7, model="gpt-4o-mini", include_reason=True)
metric.measure(test_case)

print(metric.score)
print(metric.reason)

If metric.score comes back as 0.0 with a reason string that reads like broken JSON or a truncated sentence, the actual root cause is usually one of:

• The judge model was cut off mid-response because max_tokens was too low for the model you configured, so DeepEval's internal JSON parser choked on an incomplete object.
• You passed a weaker or smaller model (some local or budget models struggle to reliably output the exact JSON schema DeepEval's prompts request).
• The retrieval_context or expected_output field was left as an empty list or None where the metric requires actual content, so the judge had nothing to compare against and defaulted to the lowest score.

Run metric.reason on the same call every time you get a suspicious 0.0 — this is the single fastest way to tell "the judge legitimately thinks the output is bad" apart from "something upstream broke."

`RuntimeError: This event loop is already running`

This one blindsides people who run DeepEval inside a Jupyter notebook, a Streamlit app, or any FastAPI/async service. The traceback looks like:

RuntimeError: asyncio.run() cannot be called from a running event loop

DeepEval uses asyncio internally to run metrics concurrently (this is how it evaluates a whole test suite in parallel instead of one test case at a time). Notebooks and async web frameworks already have an event loop running, and DeepEval trying to spin up its own nested loop collides with it.

Inside Jupyter or IPython, the standard fix is nest_asyncio:

import nest_asyncio
nest_asyncio.apply()

from deepeval import evaluate
from deepeval.metrics import FaithfulnessMetric

# now evaluate() and metric.measure() won't collide with the notebook's own loop

If you're hitting this inside a FastAPI endpoint or any service with its own async runtime, don't call the synchronous evaluate() or metric.measure() — use the async variants DeepEval ships instead:

import asyncio
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

async def run_eval(test_case: LLMTestCase):
    metric = FaithfulnessMetric(threshold=0.7)
    await metric.a_measure(test_case)
    return metric.score, metric.reason

# inside an already-running async context:
score, reason = await run_eval(test_case)

The pattern to remember: synchronous measure()/evaluate() for plain scripts and pytest, async a_measure()/a_evaluate() for anything that already owns an event loop.

`deepeval test run` hangs, then times out

You run your suite, it sits there for a while, and eventually you get something like:

httpx.ReadTimeout: The read operation timed out

or the process just never returns and you have to Ctrl+C it. Three usual suspects, roughly in order of likelihood:

• Rate limiting on the judge model provider. If you're evaluating a large dataset with concurrent metric calls, you can burn through OpenAI's requests-per-minute limit fast, and DeepEval's retry/backoff behavior can make a run feel like it's hanging when it's actually silently retrying in the background.
• Too much concurrency for your account tier. DeepEval parallelizes test cases by default. On a free-tier or low-tier API key, that concurrency itself triggers rate limits.
• A genuinely slow custom model (self-hosted via Ollama, vLLM, or similar) that just takes a long time per call, multiplied across every test case and every metric.

The fix for rate-limit-driven hangs is to cap concurrency explicitly rather than let DeepEval assume you have generous API limits:

from deepeval import evaluate
from deepeval.metrics import AnswerRelevancyMetric

evaluate(
    test_cases=my_test_cases,
    metrics=[AnswerRelevancyMetric(threshold=0.7)],
    max_concurrent=2,  # lower this if you're seeing rate-limit timeouts
)

If you're running inside pytest via deepeval test run, the equivalent lever is the -n flag DeepEval exposes for parallel workers — turn it down (or off, with -n 1) when you're debugging a hang, then turn it back up once you've confirmed the rate limit was the actual cause.

`ValidationError` and JSON decode errors from the judge model

Newer DeepEval versions use Pydantic models to validate the structured output coming back from the judge LLM, so instead of a silent 0.0, you'll sometimes see a loud and fairly intimidating error:

pydantic_core._pydantic_core.ValidationError: 1 validation error for ReasonScore
score
  Field required [type=missing, input_value={'reasoning': 'The output...'}, input_type=dict]

or, on the raw parsing side:

json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 245 (char 244)

Both mean the same underlying thing: the judge model returned text that doesn't match the exact JSON schema DeepEval's metric prompt asked for. This happens most with smaller, weaker, or heavily quantized local models that aren't reliable at strict structured output, and it happens occasionally even with capable models if temperature is set too high for a judging task (higher temperature increases the odds of the model wandering into prose instead of clean JSON).

Two fixes, and you usually want both:

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCaseParams

correctness_metric = GEval(
    name="Correctness",
    criteria="Determine whether the actual output is factually correct given the expected output.",
    evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT, LLMTestCaseParams.EXPECTED_OUTPUT],
    model="gpt-4o-mini",   # use a model reliable at structured JSON output
    threshold=0.7,
)

Set the judge model explicitly to one known to be good at instruction-following and structured output rather than leaving it on a default that might route to something weaker. If you're stuck using a local or custom model that can't reliably produce valid JSON, wrap it with a custom DeepEvalBaseLLM subclass that adds a retry-and-repair step — catch the parse failure, re-prompt with "your last response wasn't valid JSON, here's the schema again," and retry once or twice before giving up. This is more code, but it's the only reliable path when you're locked into a weaker self-hosted model for cost or privacy reasons.

Dataset and Golden mismatches: `KeyError` and missing fields

When you build test cases from a EvaluationDataset loaded from CSV, JSON, or a hosted dataset on Confident AI, a very common error is:

KeyError: 'expected_output'

or

deepeval.errors.MissingTestCaseParamsError: Unable to run metric 'Faithfulness' — 'retrieval_context' is required but was not provided on this LLMTestCase.

This happens because different metrics require different fields on LLMTestCase, and it's easy to build a dataset that has input and actual_output but forgot retrieval_context (needed for RAG metrics like FaithfulnessMetric and ContextualRecallMetric) or expected_output (needed for metrics that compare against a ground truth).

The fix is to check each metric's required parameters before you run a full suite, not after:

from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric

# FaithfulnessMetric needs: input, actual_output, retrieval_context
# AnswerRelevancyMetric needs: input, actual_output
# Check the metric docs for its required LLMTestCaseParams before building your dataset

test_case = LLMTestCase(
    input="How do I reset my password?",
    actual_output="Go to Settings > Security > Reset Password.",
    retrieval_context=["To reset your password, navigate to Settings, then Security, then select Reset Password."],
    expected_output="Navigate to Settings, then Security, then Reset Password.",
)

A good habit: build one canonical LLMTestCase with every field populated for each row in your dataset, even if a given metric doesn't need all of them. It costs a little extra dataset-prep time, but it means you can add or swap metrics later without re-auditing your entire dataset for missing fields.

Confident AI login and dataset push/pull errors

If you're using DeepEval alongside Confident AI (the hosted platform for tracking eval results over time), a common error when trying to push results is:

Error: You are not logged in. Please run `deepeval login` first.

Or, less obviously, a push that "succeeds" but shows no data on the dashboard — usually because the API key used for deepeval login belongs to a different project/organization than the one you're viewing in the browser. Double-check that the key printed during deepeval login matches the project you expect by confirming the project name shown in the CLI output against the dashboard, not just assuming they match.

For CI environments, don't run deepeval login interactively — it expects a browser flow. Instead, set the CONFIDENT_API_KEY environment variable directly as a CI secret, which DeepEval picks up non-interactively:

# example GitHub Actions step
- name: Run DeepEval suite
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
    CONFIDENT_API_KEY: ${{ secrets.CONFIDENT_API_KEY }}
  run: deepeval test run tests/

A quick triage checklist

When a DeepEval run fails and the error message isn't immediately obvious, work through these in order rather than guessing:

1. Print os.getenv("OPENAI_API_KEY") (or your provider's equivalent) right before the failing call — confirm it's actually present in this process, not just your shell.
2. Check metric.reason on any metric returning an unexpectedly low score — it tells you what the judge actually saw.
3. If you're in a notebook or async app, check whether you need nest_asyncio.apply() or the a_measure()/a_evaluate() variants instead of the sync ones.
4. If a run hangs, lower max_concurrent (or -n in pytest) before assuming something is broken — it's often just rate limiting.
5. Confirm every LLMTestCase has the fields the specific metric requires — retrieval_context, expected_output, etc. — before blaming the metric itself.
6. Pin your DeepEval version in requirements.txt or pyproject.toml so a silent upstream API change doesn't break your suite without warning.

Most DeepEval errors trace back to one of these six causes. Once you've debugged each of them once, the next one takes thirty seconds instead of thirty minutes.

Where to go deeper

DeepEval's error messages have gotten more informative with every release, but the gap between "here's a stack trace" and "here's why your evaluation pipeline is actually failing" still takes hands-on practice to close — especially once you're combining custom metrics, RAG pipelines, and CI gating in the same suite. If you want a structured, project-based walkthrough of setting up DeepEval correctly from the start — including how to design test cases, choose the right metrics for your use case, and wire evaluation into CI without the flaky failures covered here — check out the DeepEval Tutorial course on teachyou.ai, built by instructors who've hit every one of these errors in production and can show you the fix in context rather than in isolation.