Speeding Up DeepEval Test Runs: Caching and Parallelization

Why your DeepEval suite feels slower every sprint

The first time you wire DeepEval into a CI pipeline, it feels instant. You have a handful of test cases, a couple of metrics, and the whole thing finishes before your coffee gets cold. Then the project grows. Someone adds a RAG pipeline with a FaithfulnessMetric and a ContextualRelevancyMetric. Someone else adds an agent that needs ToolCorrectnessMetric alongside an LLM-judged AnswerRelevancyMetric. Six months later you have three hundred test cases, five metrics per case, and a CI job that takes twenty-five minutes to tell you whether your prompt change broke anything.

This is the point where most teams do one of two unhelpful things. They either stop running the full suite on every pull request and only run it nightly, which means regressions sit undetected for a day, or they start deleting test cases to keep the pipeline fast, which quietly erodes the thing the eval suite was supposed to protect. Neither is necessary. DeepEval test runs are slow almost entirely because of two solvable problems: redundant LLM calls that never needed to happen twice, and test execution that runs serially when it could run in parallel. Fix those two things and a twenty-five minute suite can often drop under five minutes without touching a single assertion.

This article walks through DeepEval's built-in caching layer, how to combine it with pytest-xdist for parallel execution, and the metric-level and infrastructure-level tricks that make the biggest difference once caching and parallelization are already in place.

Where the time actually goes

Before reaching for caching or parallelization, it helps to know what you're optimizing. A DeepEval test run has three main cost centers:

• LLM-judged metric evaluation. Metrics like AnswerRelevancyMetric, FaithfulnessMetric, and GEval don't just compute a number — they make one or more calls to an LLM (usually GPT-4 class or your configured judge model) to generate reasoning and a verdict. This is almost always the dominant cost, both in time and in API spend.
• The system under test. If your test case generation calls your actual RAG pipeline or agent to produce actual_output at test time, that's another round trip (or several, if there are retrieval and tool-use steps) per test case.
• Test harness overhead. pytest collection, fixture setup, and DeepEval's own bookkeeping (writing to the local cache, printing progress bars) — this is usually small but not zero at scale.

Most teams assume the bottleneck is "the LLM is slow," and that's true, but the more precise diagnosis is "the LLM is being asked the same question more times than necessary, and those questions are being asked one at a time." Caching addresses the first half. Parallelization addresses the second.

DeepEval's built-in caching

DeepEval ships with a caching mechanism that stores metric results keyed on a hash of the test case content and the metric configuration. If you re-run the same test case with the same metric and nothing about the input has changed, DeepEval can skip the LLM call entirely and return the cached verdict.

The flag that controls this is -c (or --use-cache) when running through the CLI, or use_cache=True when calling evaluate() directly:

from deepeval import evaluate
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

test_case = LLMTestCase(
    input="What is the refund window for annual plans?",
    actual_output="Annual plans can be refunded within 30 days of purchase.",
    retrieval_context=["Annual subscriptions are refundable within 30 days."],
)

metrics = [
    AnswerRelevancyMetric(threshold=0.7),
    FaithfulnessMetric(threshold=0.7),
]

evaluate(
    test_cases=[test_case],
    metrics=metrics,
    use_cache=True,
)

From the CLI it looks like this:

deepeval test run test_suite.py -c

The cache is stored locally (in a .deepeval-cache.json file by default) and keyed so that changing the input, actual_output, expected_output, retrieval_context, or the metric's threshold/model configuration invalidates just that entry, not the whole cache. This matters in CI: if you're re-running the exact same suite because a previous run flaked on network timeouts, caching means only the tests that actually failed need to hit the LLM again.

There's a subtlety worth calling out. Caching helps most in two very specific situations: re-running a suite after a flaky failure, and running the same regression suite repeatedly during local iteration where you're only changing test cases in one file. It helps far less on a genuinely fresh CI run against a new commit, because if your application's outputs change even slightly (which they will, if you changed a prompt or a model), the cache key changes and you pay the full LLM cost anyway. Caching is a defense against redundant work, not a way to avoid ever calling an LLM judge.

Making caching actually stick in CI

The most common way teams sabotage their own caching is by giving every CI run a clean filesystem. If .deepeval-cache.json gets wiped between runs, you get zero benefit from caching no matter how many times you pass -c. The fix is to persist the cache file across CI runs the same way you'd persist a dependency cache.

For GitHub Actions, that means adding a cache step keyed on something stable, like the branch name, so partial reruns on the same PR reuse prior results:

- name: Restore DeepEval cache
  uses: actions/cache@v4
  with:
    path: .deepeval-cache.json
    key: deepeval-cache-${{ github.ref }}
    restore-keys: |
      deepeval-cache-

- name: Run DeepEval suite
  run: deepeval test run test_suite.py -c

- name: Save DeepEval cache
  uses: actions/cache/save@v4
  if: always()
  with:
    path: .deepeval-cache.json
    key: deepeval-cache-${{ github.ref }}-${{ github.run_id }}

The if: always() on the save step matters — you want to persist the cache even if some tests failed, since the tests that passed still don't need to be re-judged next time. A cache that only saves on green builds throws away exactly the data you need when you're iterating on a fix for a red build.

Parallelizing with pytest-xdist

Caching removes redundant work. Parallelization is how you deal with the work that's genuinely necessary — three hundred distinct test cases each needing a fresh LLM judgment. Since DeepEval's test runner sits on top of pytest, you get parallel execution almost for free with pytest-xdist.

Install it alongside DeepEval:

pip install pytest-xdist

Then run your suite across multiple workers:

deepeval test run test_suite.py -n 4

Under the hood this is equivalent to pytest test_suite.py -n 4, splitting your test cases across four worker processes. Because LLM-judged metrics spend most of their wall-clock time waiting on network I/O rather than burning CPU, this parallelizes extremely well — you're not fighting the GIL for meaningful compute, you're just overlapping a bunch of API calls that were previously waiting on each other.

A reasonable starting point is to match worker count to your judge model provider's effective concurrency limit rather than your machine's core count. If your rate limit tier allows sixty requests a minute, spinning up sixteen workers just means fifteen of them spend their time retrying on 429s. Start conservative:

deepeval test run test_suite.py -n 4 -c

Then watch your provider's dashboard for rate-limit errors and step the worker count up or down. Most teams land somewhere between four and eight workers for a single OpenAI or Anthropic account on a standard tier — enough to meaningfully cut wall-clock time without tripping rate limits constantly.

If you want workers assigned dynamically rather than in fixed chunks (useful when some test cases are much more expensive than others, e.g., a FaithfulnessMetric case with a ten-thousand-token retrieval context next to a simple factual check), use the loadscope or loadgroup distribution modes:

deepeval test run test_suite.py -n 4 --dist loadscope

This keeps test cases grouped by module while still balancing load reasonably across workers, which avoids a scenario where one worker gets stuck with all your heaviest RAG-context test cases while the others finish early and sit idle.

Combining caching and parallelization without collisions

There's one sharp edge worth knowing about before you combine -c and -n in the same run: DeepEval's local JSON cache file is not natively designed for high-concurrency writes from multiple pytest-xdist workers. In practice this usually resolves fine because DeepEval handles the read/write locking internally, but if you see intermittent cache corruption or workers silently missing cache hits under heavy parallelization, the safest pattern is:

1. Run once with caching disabled and high parallelism to populate results fast on a fresh commit.
2. Run subsequent re-runs (e.g., retrying flaky tests) with caching enabled, since at that point you're re-running a much smaller subset and concurrent writes are less of a concern.

# First pass on a new commit — maximize parallelism, skip cache reads
deepeval test run test_suite.py -n 8

# Re-run just the failures — cache is now warm and worker count is low
deepeval test run test_suite.py -n 2 -c

This two-phase approach is more reliable in practice than trying to get maximum caching and maximum parallelism simultaneously on the very first run of a new commit, where the cache wouldn't have hits yet anyway.

Cutting cost at the metric level

Parallelization and caching change *when* LLM calls happen, not *how many* happen. If you want to reduce the actual number of judge calls, look at your metric choices themselves.

• Use a cheaper, faster judge model for metrics where it's appropriate. Not every metric needs your most expensive model as the judge. AnswerRelevancyMetric on straightforward factual QA often works fine with a smaller, faster model; reserve your top-tier judge model for metrics evaluating nuanced things like tone or multi-step reasoning correctness.

from deepeval.metrics import AnswerRelevancyMetric

metric = AnswerRelevancyMetric(
    threshold=0.7,
    model="gpt-4o-mini",
)

• Reduce redundant metrics per test case. It's common to accumulate metrics over time — someone adds FaithfulnessMetric, someone else adds a custom GEval metric that checks roughly the same thing with different wording. Audit your metric list per test suite periodically and consolidate overlapping checks. Each metric on each test case is a separate LLM call (sometimes several, since some metrics generate multiple internal reasoning steps); removing one redundant metric across three hundred test cases removes three hundred calls.

• Segment your suite into tiers. Not every test case needs to run on every commit. A pattern that works well: a small, fast "smoke" suite of the twenty or thirty most critical test cases runs on every push, gated with a tight threshold and fast judge model. The full three-hundred-case suite runs on merge to main or on a schedule, with full parallelization and caching applied.

# On every PR push — fast feedback loop
deepeval test run smoke_suite.py -n 4 -c

# On merge to main — full regression coverage
deepeval test run full_suite.py -n 8 -c

This tiering is often the single highest-leverage change a team can make, because it directly addresses the actual pain point (waiting twenty-five minutes for feedback on a PR) without touching the completeness of your full regression coverage.

Speeding up the system-under-test calls

If your test cases generate actual_output at test time by calling your live application (rather than using pre-recorded outputs), that call is often slower than the metric evaluation itself, especially for RAG pipelines with retrieval steps or agents with multiple tool calls. A few practical fixes:

• Batch your application calls outside the metric loop. Generate all actual_output values in one async batch pass before handing test cases to DeepEval's evaluate(), rather than letting DeepEval trigger your app synchronously per test case inside the pytest run.

import asyncio
from deepeval.test_case import LLMTestCase

async def generate_outputs(inputs, rag_pipeline):
    tasks = [rag_pipeline.aquery(i) for i in inputs]
    return await asyncio.gather(*tasks)

inputs = ["What is the refund policy?", "How do I reset my password?"]
outputs = asyncio.run(generate_outputs(inputs, rag_pipeline))

test_cases = [
    LLMTestCase(input=i, actual_output=o)
    for i, o in zip(inputs, outputs)
]

• Cache your application's own responses separately from DeepEval's metric cache, especially if actual_output generation involves expensive retrieval or a slow vector database. This is a cache you control and can key however makes sense for your app — by input hash, by document version, whatever fits.

• Use DeepEval's async metric methods (a_measure instead of measure) if you're evaluating outside the pytest runner in a custom script, so metric calls for different test cases can be in flight concurrently rather than sequentially awaited.

Watching for the tradeoffs

Speed work has failure modes of its own, and it's worth naming them so you don't trade a slow suite for an unreliable one.

• Over-aggressive caching can hide real regressions. If your cache key doesn't account for something that should invalidate it — say, you changed your system prompt but the test case's actual_output was pre-recorded rather than generated fresh — you can get a stale pass. Make sure caching is keyed on the actual content being judged, not just the test case's identity.
• Too much parallelism trips rate limits, and DeepEval's retry behavior on 429s can end up slower than running serially. If you crank workers up and start seeing retries dominate your run time, that's a sign to dial back, not push further.
• Tiered suites need discipline. A smoke suite that never gets updated as your app evolves stops representing real risk areas. Revisit which test cases live in the fast tier every so often, ideally whenever you ship a feature that touches a part of the system the smoke suite doesn't currently cover.

A practical checklist for your next optimization pass

• 1. Turn on use_cache=True (or -c) and confirm your CI persists the cache file across runs instead of starting fresh every time.
• 2. Add pytest-xdist and start with -n 4, watching for rate-limit errors before scaling up further.
• 3. Split your suite into a fast smoke tier for every PR and a full tier for merges or nightly runs.
• 4. Audit your metrics for redundancy and consider a cheaper judge model for straightforward checks.
• 5. Move actual_output generation to a batched async pass outside the metric evaluation loop.
• 6. Re-measure. Track wall-clock time and LLM spend before and after each change so you know which lever actually moved the needle for your specific suite.

None of these changes require rewriting your test cases or lowering your evaluation standards. They're infrastructure changes sitting underneath assertions that stay exactly as rigorous as they were before. The suite gets faster; the bar for correctness doesn't move.

If you want a guided, hands-on walkthrough of building and scaling a DeepEval suite from scratch — including caching, parallel CI setups, and tiered test strategies applied to a real RAG project — check out the DeepEval Tutorial course on TeachYou.ai. It covers exactly the workflow described here, end to end, with a project you build alongside the lessons.