DeepEval Command Line Interface: A Practical Reference

Why The DeepEval CLI Deserves Its Own Cheat Sheet

Most teams meet DeepEval through its Python API: import an LLMTestCase, pick a metric, call evaluate(). That's the right starting point, but it hides the fact that DeepEval ships a genuinely useful command line interface that changes how you run, monitor, and scale evaluations once you move past a single notebook. The CLI is what turns a one-off evaluation script into something you can run in a pre-commit hook, wire into GitHub Actions, or fire off from a terminal while iterating on a prompt.

The problem is that most of what's written about DeepEval focuses on metric definitions and dataset construction, and the CLI gets a passing mention at best. That's a shame, because commands like deepeval test run, deepeval login, and deepeval view carry real weight in day-to-day workflows: they control parallelization, caching, Confident AI sync, and how failures get reported back to you. If you're building or evaluating LLM applications and you've only ever run pytest against your eval file, you're leaving useful tooling on the table.

This article is a practical reference to the DeepEval command line interface. We'll go through installation, authentication, the core test run command and its flags, dataset and result management commands, caching behavior, and how to slot all of this into a CI pipeline. Wherever it helps, we'll show the actual command and explain what changes when you flip a flag.

Installing DeepEval And Confirming The CLI Is Live

DeepEval installs from PyPI like any other Python package, and the CLI binary comes bundled with it — there's no separate install step.

pip install -U deepeval

Once installed, confirm the CLI entry point resolves and check the version you're on:

deepeval --version

If you're working inside a virtual environment (which you should be), make sure that environment is active before running any deepeval command, since the CLI is registered as a console script tied to that Python installation. A common early mistake is installing DeepEval in one virtualenv and then running deepeval from a shell where a different (or no) virtualenv is active — you'll get a "command not found" or, worse, a stale version from an old global install.

To see everything the CLI exposes at a glance, run it without a subcommand or with --help:

deepeval --help

This prints the full list of top-level commands — test, login, logout, view, dataset, and others depending on your installed version — along with a one-line description of each. Treat this as your first stop whenever you upgrade DeepEval, since new CLI subcommands get added between releases and --help output stays accurate even when docs lag behind.

Authenticating With `deepeval login`

A meaningful chunk of the CLI's value comes from Confident AI, the hosted platform DeepEval integrates with for storing test run history, visualizing metric trends over time, and sharing results with teammates who don't want to read raw JSON. None of that requires Confident AI for local evaluation to work — DeepEval runs perfectly well fully offline — but if you want run history that survives beyond your terminal scrollback, deepeval login is the command that wires it up.

deepeval login

Running this opens a browser flow (or prints a URL you paste into a browser if you're on a headless box) to authenticate and generate an API key, which the CLI then stores locally so subsequent commands pick it up automatically. If you're running this in a CI environment where there's no browser to pop open, you can instead pass the API key directly as a flag or environment variable:

deepeval login --confident-api-key "your-api-key-here"

or, equivalently, export it before invoking any other command:

export CONFIDENT_API_KEY="your-api-key-here"

This second pattern is what you want in GitHub Actions or any pipeline where secrets come from a vault or repo secret store rather than an interactive prompt. Once authenticated, every deepeval test run invocation automatically pushes results to your Confident AI project, and you'll see a link printed in the terminal output pointing straight to the run's dashboard page.

To confirm you're actually logged in (useful when debugging a CI job that silently isn't syncing), check for the stored key or simply run a test and watch for the "View results on Confident AI" link in the output. If that link is missing, the run executed but nothing got pushed — which usually means the API key wasn't picked up.

To remove stored credentials, for example when rotating a key or switching between personal and team accounts:

deepeval logout

The Core Command: `deepeval test run`

This is the command you'll type more than any other. It's DeepEval's wrapper around pytest, purpose-built for evaluation files, and it accepts a path to a Python test file containing your LLMTestCase and metric definitions.

deepeval test run test_chatbot.py

Under the hood, this is literally running pytest against your file, but with DeepEval's plugin injected so that metrics get evaluated, scores get computed, and a readable pass/fail summary table gets printed at the end instead of raw pytest assertion errors. Because it's pytest under the hood, standard pytest flags still apply — you can target a specific test function, use -k to filter by name, or add -v for verbose output.

deepeval test run test_chatbot.py -k "test_summarization"

A minimal test file looks like this:

from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_answer_relevancy():
    test_case = LLMTestCase(
        input="What is the capital of France?",
        actual_output="The capital of France is Paris.",
        retrieval_context=["Paris is the capital and most populous city of France."]
    )
    metric = AnswerRelevancyMetric(threshold=0.7)
    assert_test(test_case, [metric])

Run it, and the CLI prints a per-test-case table showing the metric name, the score, whether it passed the threshold, and the reasoning the underlying judge model produced. If a case fails, the reason string is usually specific enough to tell you exactly what went wrong — a retrieved chunk that didn't support the claim, a summary that dropped a key fact, or an answer that drifted off-topic.

Parallelizing Runs With `-n`

Evaluation runs are slow by nature — every metric that uses an LLM-as-judge (which is most of DeepEval's built-in metrics) makes at least one API call per test case, sometimes several if the metric decomposes the output into claims or verdicts first. Once your test suite grows past a handful of cases, sequential execution turns a quick sanity check into a coffee-break-length wait.

DeepEval solves this with pytest-xdist support baked into deepeval test run, exposed through the -n flag for parallel workers:

deepeval test run test_chatbot.py -n 4

This spins up four parallel workers, each handling a subset of test cases, roughly quartering wall-clock time on a large suite (actual speedup depends on your API rate limits — if you're hammering an LLM provider that throttles you, adding workers won't help and may trigger rate-limit errors instead). A sensible pattern is to start with a small worker count, watch for 429 errors from your model provider, and dial -n up or down based on what your account's rate limits tolerate.

deepeval test run test_chatbot.py -n 8

For genuinely large regression suites — the kind you'd run nightly rather than on every commit — pushing worker count higher combined with async metric execution (which DeepEval's metrics support natively) is the difference between a 45-minute run and a 6-minute one.

Caching With `-c` And Re-Running Only Failures

One of the more underrated flags is -c, which enables DeepEval's caching layer. When caching is on, DeepEval stores the metric results for each test case keyed by a hash of the test case content plus the metric configuration. If you rerun the same file and nothing about a given case or its metric changed, the cached score gets reused instead of making a fresh (and costly) LLM call.

deepeval test run test_chatbot.py -c

This matters most when you're iterating on one part of a large test file. Say you have 200 test cases and you're only actively debugging the 10 that cover a new feature — with caching on, reruns skip re-evaluating the 190 unaffected cases and only pay the API cost for the ones that actually changed.

Pair this with -r or the retry-specific behavior for re-running failing cases after a fix. A common loop during active development looks like this: run the full suite, see three failures, fix the prompt template, then rerun just the failing cases to confirm before doing a full clean run:

deepeval test run test_chatbot.py -c -n 4

Combining -c and -n is the standard pattern for any repo with a nontrivial evaluation suite — caching cuts redundant cost, parallelization cuts wall-clock time, and together they make it realistic to run evaluations on every pull request rather than only on a weekly cadence.

Working With Datasets From The CLI

DeepEval treats evaluation datasets as first-class objects, and if you're pulling test cases from Confident AI rather than hardcoding them in a Python list, the deepeval dataset family of commands is where that happens.

Pull a dataset that was created and curated on Confident AI's platform down into a local file:

deepeval dataset pull --alias "customer-support-golden-set"

This fetches the named dataset and materializes it locally so your test file can load it without needing network access to Confident AI at test-collection time — only at evaluation time do the underlying metrics reach out to your judge LLM. Once pulled, referencing it in a test file typically looks like loading an EvaluationDataset and iterating its test_cases in a parametrized pytest test:

from deepeval.dataset import EvaluationDataset

dataset = EvaluationDataset()
dataset.pull(alias="customer-support-golden-set")

for test_case in dataset.test_cases:
    print(test_case.input)

The CLI pull step and the Python .pull() method achieve the same thing — the CLI form is convenient when you want to snapshot a dataset into your repo or CI cache ahead of time, decoupling the "get data" step from the "run evaluation" step so a flaky network call to fetch data doesn't fail your test job halfway through.

Viewing Results With `deepeval view`

Once a run finishes and results are synced to Confident AI, you don't have to leave your terminal to jump into the dashboard. The view command opens the most recent test run's results page directly in your default browser:

deepeval view

This is a small quality-of-life command, but it removes friction from a workflow that otherwise involves scrolling back through terminal output to find the printed URL, copying it, and pasting it into a browser tab. If you're pairing with a teammate over a screen share, deepeval view right after a run finishing is the fastest way to get both of you looking at the same score breakdown, per-case reasoning, and historical trend chart.

Setting Up A Confident AI Project From The CLI

If you're starting a brand-new evaluation project and want it tracked on Confident AI from day one, initialization typically happens through a setup command that links your local repo to a hosted project:

deepeval init

This walks you through naming the project, confirming the API key is available (prompting a login if it isn't), and writing a small local config so subsequent deepeval test run invocations know which project to push results into without you passing extra flags each time. For teams running DeepEval across multiple repositories — say, one repo per microservice that each has its own LLM-backed feature — this per-repo project linkage keeps evaluation history cleanly separated instead of everything landing in one undifferentiated project.

Wiring `deepeval test run` Into CI

The whole point of a CLI-first workflow is that it drops into CI without translation. A GitHub Actions job for running evaluations on every pull request looks close to this:

name: LLM Evaluation

on:
  pull_request:
    branches: [main]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install dependencies
        run: pip install -U deepeval

      - name: Run DeepEval test suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          CONFIDENT_API_KEY: ${{ secrets.CONFIDENT_API_KEY }}
        run: deepeval test run tests/test_llm_app.py -n 4 -c

A few details matter here. First, the API key for whichever LLM you're using as a judge (OpenAI, Anthropic, or a self-hosted model endpoint) needs to be available as an environment variable — DeepEval's metrics read it the same way they would locally. Second, setting CONFIDENT_API_KEY as a repo secret means every CI run automatically syncs to Confident AI without an interactive deepeval login, which wouldn't work in a non-interactive runner anyway. Third, because deepeval test run is a pytest wrapper, a failing metric threshold causes the job step to exit non-zero, which fails the CI check and blocks the merge — exactly the gate you want for a prompt or model change that regresses quality.

For teams that only want evaluation to run on specific paths (say, only when prompt templates or the RAG retrieval logic change, not on every doc update), scope the workflow trigger with a paths: filter under pull_request so evaluation cost doesn't get spent on unrelated changes.

Troubleshooting Common CLI Friction Points

A short list of the issues that come up most often when teams first wire the CLI into real workflows:

• `deepeval` command not found after pip install — almost always a virtualenv mismatch. Confirm which deepeval points inside your active environment's bin directory, not a global or different venv's path.
• Test run hangs or times out — usually a judge-model API call stalling. Check that your LLM provider's API key is valid and that you're not silently rate-limited; add -n 1 temporarily to isolate whether parallelization is compounding a rate-limit issue.
• Results aren't showing on Confident AI — check that CONFIDENT_API_KEY is actually set in the shell running the command, not just exported in a different terminal session or a .env file your shell never sourced.
• Cached results feel stale after a prompt change — caching keys off test case content and metric config, but if your application code changed (say, a new system prompt) without changing the LLMTestCase fields DeepEval hashes, you may need to clear the cache explicitly or rerun without -c once to force fresh scores.
• Flaky pass/fail on the same input — this is inherent to LLM-as-judge metrics, not a CLI bug. Judge models have some run-to-run variance; setting a slightly lower threshold or using strict_mode deliberately, rather than chasing perfect determinism, is usually the right response.

Most of these resolve in under five minutes once you know where to look, but they're exactly the kind of friction that makes a first CI integration take an afternoon instead of ten minutes if you're going in blind.

Bringing It Together

The DeepEval CLI is small in surface area but disproportionately useful once your evaluation suite grows past a single script you run by hand. deepeval login gets your runs synced to a dashboard your whole team can see. deepeval test run — with -n for parallelization and -c for caching — is the command that scales from a two-case smoke test to a two-hundred-case regression suite without becoming unbearably slow or expensive. deepeval dataset pull decouples your golden test data from your test code, and deepeval view closes the loop by getting you from a finished run to a browser dashboard in one keystroke.

None of this replaces understanding what the metrics themselves measure — that's still where the real evaluation judgment lives. But the CLI is the layer that determines whether your evaluations actually get run consistently, whether failures block bad merges, and whether your team can see quality trends over time instead of re-litigating "is the bot good" from scratch every sprint. If you're building LLM applications and evaluation still feels like a manual, occasional chore, wiring the CLI into your existing test and CI setup is usually the fastest way to change that.

If you want a structured, hands-on walkthrough of DeepEval — from your first LLMTestCase through custom metrics, dataset curation, and full CI integration — our DeepEval Tutorial course on teachyou.ai covers the CLI and the underlying evaluation concepts in depth, with real project examples you can adapt directly into your own pipeline.