DeepEval Confident AI Platform: When to Go Beyond Open Source

The question every team hits eventually

You start with DeepEval because it's free, it's Pythonic, and it runs your LLM evals right inside pytest like any other test suite. For weeks this feels perfect. Then one Tuesday your product manager asks, "which prompt version caused the regression in our summarization scores last month?" and you realize the answer is scattered across a dozen CI logs, a spreadsheet someone abandoned in March, and your own memory. That's usually the moment teams start looking at Confident AI, the hosted platform built by the same team behind the open source DeepEval library.

This article is not a sales pitch. It's a breakdown of what DeepEval's open source core actually does, what Confident AI adds on top, and — more importantly — the honest tradeoffs of paying for a platform versus stitching together your own dashboards. If you're evaluating whether to stay open source or go hosted, this should give you enough to make that call without a sales call.

What DeepEval actually is, in one paragraph

DeepEval is an open source Python framework for evaluating LLM outputs. It ships pre-built metrics — answer relevancy, faithfulness, contextual precision and recall for RAG pipelines, hallucination detection, toxicity, bias, and a general-purpose GEval metric that lets you define a custom rubric in plain English and have an LLM judge score against it. It integrates with pytest, so an eval suite reads like a normal test file:

from deepeval import assert_test
from deepeval.metrics import GEval, FaithfulnessMetric
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

def test_support_bot_response():
    test_case = LLMTestCase(
        input="How do I reset my password?",
        actual_output="Go to Settings > Security > Reset Password.",
        retrieval_context=[
            "To reset a password, navigate to Settings, then Security, then click Reset Password."
        ],
    )

    correctness = GEval(
        name="Correctness",
        criteria="Determine if the actual output correctly answers the input question, using only the retrieval context as ground truth.",
        evaluation_params=[
            LLMTestCaseParams.INPUT,
            LLMTestCaseParams.ACTUAL_OUTPUT,
            LLMTestCaseParams.RETRIEVAL_CONTEXT,
        ],
        threshold=0.7,
    )

    faithfulness = FaithfulnessMetric(threshold=0.7)

    assert_test(test_case, [correctness, faithfulness])

Run deepeval test run test_support_bot.py and you get pass/fail output with scores and reasoning per metric. This is genuinely useful on day one, and it's completely free — no account, no API key beyond your LLM provider, no data leaving your machine except what you send to the judge model itself.

Confident AI is the commercial layer wrapped around this same library. The metrics, the test case objects, the assert_test function — all of that stays open source and MIT-licensed. What Confident AI adds is a hosted dashboard, dataset management, regression tracking across runs, team collaboration, and production monitoring. You keep writing the same DeepEval code; you just add an API key and your results start flowing to a web UI instead of (or in addition to) your terminal.

Why local-only evals stop scaling

The pytest-in-terminal workflow is fantastic for a single developer iterating on a prompt. It falls apart in three predictable ways once a team or a product grows around it.

• No historical record. Terminal output disappears the moment your CI job ends unless you're piping it somewhere. You can't answer "what was our faithfulness score three weeks ago" without re-running old code against old data, assuming you even kept both.
• No shared visibility. A pytest failure in CI is visible to whoever's watching that pipeline. It is not visible to the PM, the founder, or the support team who actually care whether the bot is hallucinating less this week than last week.
• No dataset versioning. Most teams' eval datasets live in a JSON file or a notebook cell, edited in place. Nobody can tell you which version of the "golden set" produced a given score, which makes every eval result slightly untrustworthy.

None of these are DeepEval's fault — they're what happens to any test suite that isn't paired with a place to store and visualize results over time. Confident AI exists specifically to solve this class of problem, and it's worth being precise about which of your pain points it actually addresses versus which ones you could solve yourself with far less commitment.

What Confident AI adds, concretely

Once you set CONFIDENT_API_KEY as an environment variable and log in via the DeepEval CLI, every deepeval test run automatically pushes results to your Confident AI project. Concretely, that unlocks:

• A test run history with diffs. Every run is stored, timestamped, and tied to the git commit or tag you were on (if you pass it in). You can open two runs side by side and see exactly which test cases flipped from pass to fail.
• Dataset hosting and versioning. Instead of a JSON file in your repo, your golden datasets live in Confident AI with version history. You can edit a test case in the UI, and someone with edit access other than you can propose changes without touching your codebase.
• Metric-level regression alerts. You can configure a Slack or email alert when a specific metric drops below a threshold across runs, rather than relying on someone actually reading CI logs.
• Production tracing. Beyond offline evals, Confident AI can ingest live traces from your deployed application (via deepeval's tracing integration or OpenTelemetry) and run the same metrics against real user traffic asynchronously, then surface it as a monitoring dashboard.
• Human review workflows. A human-in-the-loop annotation interface where domain experts can label LLM outputs as correct/incorrect and feed that back into few-shot examples or fine-tuning datasets.
• Team and role management. Multiple engineers and non-engineers get scoped access to the same evaluation project, which matters the moment your eval process needs input from someone who doesn't write Python.

None of this changes what a metric computes. FaithfulnessMetric gives the same score whether you're logged into Confident AI or running fully offline. What changes is what happens to that score after it's computed — where it's stored, who can see it, and whether it's compared against anything.

The honest open source case

Here's where I'll push back on the "just upgrade to hosted" instinct, because for a large slice of teams, staying open source is the right call, not the compromise call.

If you're a solo developer or a small team validating a prototype, you do not need dataset versioning UI, alerting, or role-based access — you need to know if your RAG pipeline's context precision is above some threshold before you ship a demo. A local pytest run and a .json file checked into your repo does that job completely. Every dollar and every minute spent configuring a hosted dashboard at that stage is a dollar not spent talking to users.

There's also a control argument. Running evals fully offline means your test data — which might include real (if scrubbed) user queries, proprietary prompts, or internal documentation used as retrieval context — never leaves your infrastructure except to whichever LLM API you're using as a judge. Some regulated environments (healthcare, finance, government contractors) will have review processes that make adding another third-party data processor a genuine cost in time and legal review, even if Confident AI's security posture is fine. If your compliance team needs to sign off on a new vendor before anyone on engineering can use it, that friction is real and worth weighing against the convenience.

And frankly, you can build a decent chunk of "hosted platform" yourself with tools you likely already have:

import json
from datetime import datetime, timezone
from deepeval import evaluate
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

def run_and_log(test_cases, log_path="eval_history.jsonl"):
    metric = FaithfulnessMetric(threshold=0.7)
    results = evaluate(test_cases, [metric])

    record = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "scores": [
            {"input": tc.input, "score": r.metrics_data[0].score}
            for tc, r in zip(test_cases, results.test_results)
        ],
    }

    with open(log_path, "a") as f:
        f.write(json.dumps(record) + "\n")

    return results

Append that JSONL to a bucket, point a simple dashboard (even a Streamlit app or a Grafana panel reading from Postgres) at it, and you have historical tracking without a subscription. It's not as polished, and you're now maintaining infrastructure instead of shipping product — but it's a legitimate middle path many teams use for a year or more before paying for anything.

When hosted actually pays for itself

The calculation flips once a few specific things become true simultaneously, and it's worth naming them precisely rather than vaguely gesturing at "scale."

1. More than one person needs to see eval results and isn't a developer. The moment a PM, founder, or support lead needs visibility without asking an engineer to paste terminal output into Slack, you're paying an invisible tax in interruptions and delayed decisions. A shared dashboard removes that tax immediately.
2. You're shipping prompt or model changes weekly or faster. At that cadence, "did this regress anything" needs to be answerable in seconds by comparing two run IDs, not by re-running old commits.
3. You have production traffic you want to monitor, not just offline test sets. Offline evals catch what you thought to test for. Production monitoring catches the failure modes you didn't anticipate — and building your own trace ingestion, sampling, and async metric scoring pipeline is a real engineering project, not a weekend script.
4. Your eval datasets are living documents, not fixtures. If subject matter experts are actively curating and expanding your golden set, a UI for that beats someone editing raw JSON and hoping they don't break the schema.
5. You need an audit trail for compliance or stakeholder reporting, where "we have logs somewhere" isn't good enough and you need a clean historical view you can screenshot for a board deck or an audit.

If two or more of these are true for your team right now, the subscription cost is very likely smaller than the engineering time you'd spend rebuilding a worse version of the same thing. If none of them are true yet, you're probably paying for headroom you don't need this quarter.

A migration path that doesn't lock you in

One thing worth calling out explicitly: moving from local-only DeepEval to Confident AI is not a rewrite. Because the metrics and test case objects are identical between the open source library and the hosted platform, adopting Confident AI is closer to flipping a switch than migrating a system.

import deepeval
deepeval.login_with_confident_api_key("your-api-key-here")

# Everything below is unchanged from your existing test suite
from deepeval import assert_test
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase

def test_existing_case():
    test_case = LLMTestCase(
        input="What's your refund policy?",
        actual_output="Refunds are processed within 5-7 business days of the return being received.",
        retrieval_context=["Our refund policy states returns are processed within 5-7 business days upon receipt."],
    )
    assert_test(test_case, [FaithfulnessMetric(threshold=0.8)])

The reverse is also true, and this matters more than most teams realize before they commit: if you outgrow the hosted platform, decide the pricing no longer fits your usage, or need to move to a fully offline environment for compliance reasons, your test code doesn't change. You remove the login call, keep running deepeval test run locally, and lose the dashboard, history, and alerting — but not your actual evaluation logic. That's a meaningfully different lock-in profile than, say, migrating off a proprietary eval DSL that only runs inside one vendor's platform. The open core model here is doing real work for your negotiating position, not just your wallet.

Practical decision framework

If you want a shortcut instead of re-reading the sections above, here's the compressed version:

• Stay open source if: you're pre-product-market-fit, a single engineer owns evals end to end, your eval cadence is weekly-or-slower, and nobody outside engineering needs to see the numbers.
• Go hosted if: eval results need to reach non-engineers regularly, you're shipping fast enough that manual regression comparison is a bottleneck, you want production trace monitoring without building it, or you're accumulating a real golden dataset that deserves versioning and review workflows.
• Consider the middle path if: you like the idea of history and dashboards but aren't ready to add a vendor — a JSONL log plus a lightweight internal dashboard covers 60-70% of the value for a fraction of the commitment, at the cost of your own maintenance time.

There's no wrong answer here in the abstract — only a wrong answer for where your team actually is right now. The mistake to avoid is either extreme: staying fully manual long after multiple stakeholders need visibility, or paying for a full platform before you have more than one person and one repo touching evals.

Closing thoughts

DeepEval's open source core is not a stripped-down trial version of something better — it's a complete, production-capable evaluation library on its own, and plenty of serious teams never need more than pytest and a metrics import. Confident AI is best understood not as "DeepEval but paid" but as a separate product solving a separate problem: the organizational problem of storing, comparing, sharing, and monitoring eval results over time, across people, across deployments. Whether you need that product depends entirely on whether that organizational problem already exists for you, or is still hypothetical.

If you're building RAG pipelines, agents, or any LLM-backed feature and want to get hands-on with writing real metrics, structuring test cases, and deciding for yourself where the open source line should sit for your project, our DeepEval Tutorial course walks through exactly this — from your first assert_test to deciding if and when a hosted platform earns its place in your stack.