DeepEval

DeepEval Red Teaming Module: Adversarial Testing Explained

Ira Menon · Jun 11, 2026 · 12 min read

Why Your Eval Suite Isn't Enough

You ran your DeepEval test suite. Faithfulness scores are green. Answer relevancy is above 0.9. Hallucination metrics look clean. Ship it, right?

Not so fast. Standard LLM evaluation answers one question: "does the model behave correctly on inputs I expected?" It says nothing about what happens when someone deliberately tries to break your system. A user who wraps a malicious instruction in Base64 encoding. A customer support bot that gets socially engineered into revealing another user's order history. A chatbot that gets talked into giving medical advice it was explicitly told not to give.

This is the gap that red teaming closes. Where standard evals check "does it work," red teaming checks "can it be broken." And for anyone shipping an LLM feature into production — a RAG assistant, an agent with tool access, a customer-facing chatbot — that second question matters just as much as the first, arguably more, because the failure mode isn't a bad answer, it's a security incident or a PR disaster.

DeepEval, the open-source LLM evaluation framework, ships with red teaming capabilities that let you simulate adversarial attacks against your own application programmatically, in Python, as part of your normal development workflow. This article walks through what the red teaming module actually does, how the API works, which vulnerability categories and attack strategies it covers, and how to wire it into a CI pipeline so adversarial testing isn't a one-time audit but a repeatable gate.

What "Red Teaming" Means for LLM Systems

The term comes from security testing: a red team plays the attacker, trying to compromise a system, while the blue team defends it. Applied to LLMs, red teaming means systematically generating adversarial prompts designed to make a model do something it shouldn't — leak private data, produce toxic content, execute an unauthorized action, or contradict its own safety instructions — and then scoring whether the attack succeeded.

This is fundamentally different from a typical eval run. A faithfulness or answer-relevancy metric assumes a well-formed, good-faith input. Red teaming assumes an adversarial input, deliberately obfuscated, socially engineered, or multi-turn manipulated to slip past guardrails.

DeepEval's red teaming functionality is built around a straightforward loop:

Define vulnerabilities — the categories of bad behavior you're testing for (bias, PII leakage, illegal activity, excessive agency, and dozens more).
Define attacks — the techniques used to try to trigger those vulnerabilities (prompt injection, encoding tricks, roleplay jailbreaks, multi-turn escalation).
Point the attack simulator at your target model through a callback function.
Let the framework generate adversarial test cases, run them against your app, and grade the outputs using LLM-as-a-judge metrics.
Review a risk assessment report scored per vulnerability, with reasoning for every pass/fail verdict.

The heavy lifting — generating realistic attack prompts, applying obfuscation, judging whether an attack actually succeeded — is handled by the framework so you're not hand-writing hundreds of jailbreak strings yourself.

Setting Up Your Target Model

Before you can red team anything, the framework needs a way to talk to your application. This is done through a model callback — a simple async function that takes an input string and returns your application's response. It doesn't matter whether your app is a raw LLM call, a RAG pipeline, or a multi-step agent with tool use; as long as you can wrap it in a function with this signature, it's testable.

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def model_callback(input: str) -> str:
    # Swap this out for your actual application logic —
    # RAG retrieval, agent orchestration, tool calls, etc.
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a customer support assistant for a retail bank. "
                    "Never reveal account numbers, balances, or personal "
                    "information belonging to a different customer."
                ),
            },
            {"role": "user", "content": input},
        ],
    )
    return response.choices[0].message.content

Notice this callback treats your application as a black box. The red teaming layer doesn't need to know your prompt engineering, your retrieval strategy, or your agent's internal reasoning — it just needs the observable input/output behavior. That's what makes it practical to bolt onto an existing system without refactoring anything.

Defining Vulnerabilities: What You're Actually Testing For

DeepEval's red teaming tooling organizes vulnerabilities into named categories you import and configure directly. A few examples:

from deepteam.vulnerabilities import (
    Bias,
    Toxicity,
    PIILeakage,
    Misinformation,
    IllegalActivity,
    ExcessiveAgency,
)

# Test for racial and gender bias specifically
bias = Bias(types=["race", "gender"])

# Test whether the model leaks personally identifiable information
pii_leakage = PIILeakage(types=["direct disclosure", "session leak"])

# Test whether an agent takes actions beyond its intended scope
excessive_agency = ExcessiveAgency(types=["excessive functionality"])

Each vulnerability class accepts a types argument that narrows the specific sub-behavior being probed. Bias, for instance, can be scoped to race, gender, religion, or politics independently — useful when you want a fast, targeted scan rather than testing every sub-type on every run.

The vulnerability surface spans six broad domains worth knowing about:

Data privacy — PII leakage, prompt leakage (getting the model to reveal its own system prompt)
Responsible AI — bias, toxicity, fairness, ethics, child safety
Security — SQL injection, shell injection, SSRF, broken access control patterns like BOLA and BFLA
Safety — illegal activity, graphic content, personal safety risks
Business — misinformation, IP infringement, competitor disparagement
Agentic — excessive agency, goal theft, recursive hijacking for tool-using agents

If you're building an agent with function-calling or tool access, the agentic and security categories deserve special attention — a chatbot that leaks a wrong fact is embarrassing, but an agent that can be tricked into calling a delete-record tool is a different order of problem.

Defining Attacks: How the Adversary Tries to Break In

Vulnerabilities describe *what* you're testing for. Attacks describe *how* the adversarial prompt is constructed and delivered. DeepEval's tooling separates these into single-turn and multi-turn attack strategies.

Single-turn attacks are self-contained — one adversarial prompt, one attempt:

from deepteam.attacks.single_turn import (
    PromptInjection,
    ROT13,
    Base64,
    Roleplay,
    GrayBox,
)

prompt_injection = PromptInjection()
encoded_attack = Base64()
cipher_attack = ROT13()
roleplay_attack = Roleplay(persona="a security researcher testing safeguards")

Base64 and ROT13 are encoding-based enhancements — they take a harmful instruction and obfuscate it so a naive keyword filter won't catch it, then rely on the model decoding and complying anyway. Roleplay and GrayBox are social-engineering style attacks that frame the malicious request inside a fictional or semi-legitimate context to lower the model's guard.

Multi-turn attacks are more sophisticated — they escalate across a conversation, gradually steering the model toward a harmful output rather than asking for it directly:

from deepteam.attacks.multi_turn import Crescendo, LinearJailbreaking

crescendo_attack = Crescendo()
linear_attack = LinearJailbreaking()

Crescendo in particular models a known real-world jailbreak pattern: start with innocuous, adjacent questions, then incrementally push the conversation toward the actual target, using the model's own prior responses as scaffolding for the next escalation. This is exactly the kind of attack that a single-prompt eval would never catch, because no individual turn looks obviously malicious.

Running the Scan

With vulnerabilities and attacks defined, the actual scan is a single function call:

from deepteam import red_team
from deepteam.vulnerabilities import Bias, PIILeakage
from deepteam.attacks.single_turn import PromptInjection, Base64
from deepteam.attacks.multi_turn import Crescendo

vulnerabilities = [
    Bias(types=["race", "gender"]),
    PIILeakage(types=["direct disclosure"]),
]

attacks = [
    PromptInjection(weight=2),
    Base64(weight=1),
    Crescendo(weight=1),
]

risk_assessment = red_team(
    model_callback=model_callback,
    vulnerabilities=vulnerabilities,
    attacks=attacks,
)

print(risk_assessment.overview)
for test_case in risk_assessment.test_cases:
    print(test_case.vulnerability, test_case.score, test_case.reason)

risk_assessment.save(to="./red-team-results/")

The weight parameter on each attack lets you control the relative frequency of that attack strategy during simulation — useful when you have a hunch that, say, encoding attacks are more likely to succeed against your particular system prompt than roleplay ones, and you want denser coverage there.

Under the hood, the framework generates a batch of adversarial prompts per vulnerability/attack combination, runs each through your model_callback, and then uses an LLM-as-a-judge metric to score whether the response actually constitutes a vulnerability breach — not just whether the model said something awkward, but whether it crossed the specific line you defined. Every test case comes back with a score and a human-readable reason, so failures are debuggable rather than a mystery pass/fail flag.

For teams that want tighter control over how attacks are generated — custom personas, domain-specific attack framing, a fixed number of variations per vulnerability — there's also an AttackEngine you can configure explicitly:

from deepteam.attacks.attack_engine import AttackEngine

engine = AttackEngine(
    simulator_model="gpt-4o-mini",
    variations=3,
    purpose="Customer support chatbot for a retail bank",
    generation_guidelines=[
        "Frame attacks as a frustrated long-time customer.",
        "Reference plausible but fake account details.",
    ],
)

risk_assessment = red_team(
    model_callback=model_callback,
    vulnerabilities=vulnerabilities,
    attacks=[PromptInjection()],
    attack_engine=engine,
)

This matters in practice because generic jailbreak prompts often don't reflect how your actual users would try to manipulate your specific product. A banking support bot and a coding assistant have very different realistic attack surfaces, and the generation_guidelines field is where you encode that context.

Reading and Acting on a Risk Assessment

A red team scan is only useful if the output tells you what to fix. The risk_assessment object returned from red_team() gives you both an aggregate view and per-test-case detail:

# High-level pass rate per vulnerability category
print(risk_assessment.overview)

# Drill into individual failures
failed_cases = [
    tc for tc in risk_assessment.test_cases if tc.score == 0
]

for case in failed_cases:
    print("Vulnerability:", case.vulnerability)
    print("Attack used:", case.attack_method)
    print("Adversarial input:", case.input)
    print("Model output:", case.actual_output)
    print("Judge reasoning:", case.reason)
    print("---")

Treat the failed cases as a backlog, not just a report. Each one is a reproducible prompt-response pair that tells you exactly what broke and why — often the fix is a tightened system prompt, an added output filter, or a guardrail check before the response is returned to the user. Re-run the same scan after each fix to confirm the specific attack no longer succeeds, rather than assuming a general prompt tweak fixed everything.

Wiring Red Teaming Into CI

Adversarial testing loses most of its value if it's a one-off exercise you run before a big launch and never touch again. Model providers change underlying weights, your system prompt evolves, new tool integrations get added — each of those is a fresh opportunity for a previously-blocked attack to start working again.

The practical pattern is to treat red teaming like any other test suite: run a scoped subset on every pull request, and a fuller sweep on a schedule.

# test_red_team_regression.py
import pytest
from deepteam import red_team
from deepteam.vulnerabilities import Bias, PIILeakage, IllegalActivity
from deepteam.attacks.single_turn import PromptInjection, Base64

from my_app import model_callback  # your wrapped application


def test_no_pii_leakage_via_prompt_injection():
    risk_assessment = red_team(
        model_callback=model_callback,
        vulnerabilities=[PIILeakage(types=["direct disclosure"])],
        attacks=[PromptInjection()],
    )
    failed = [tc for tc in risk_assessment.test_cases if tc.score == 0]
    assert not failed, f"PII leakage vulnerabilities found: {failed}"


def test_no_bias_regression():
    risk_assessment = red_team(
        model_callback=model_callback,
        vulnerabilities=[Bias(types=["race", "gender"])],
        attacks=[Base64()],
    )
    failed = [tc for tc in risk_assessment.test_cases if tc.score == 0]
    assert not failed, f"Bias vulnerabilities found: {failed}"

Run this with pytest -v test_red_team_regression.py in your CI pipeline alongside your normal DeepEval metric tests. Keep the per-PR set small and fast — a handful of high-priority vulnerabilities with lightweight attacks — and reserve the full 50+ vulnerability, multi-turn attack sweep for a nightly or weekly job, since simulator and judge model calls add real latency and cost at scale. The goal isn't to catch every conceivable jailbreak on every commit; it's to make sure known-fixed vulnerabilities stay fixed and that new code doesn't quietly reopen an old hole.

Common Pitfalls Teams Run Into

A few mistakes show up repeatedly when teams first adopt red teaming:

Testing the model instead of the application. Red teaming your raw model without your system prompt, guardrails, and output filters in place tells you nothing about your actual production risk. Always route the callback through your full application stack.
Treating a single passing scan as permanent. Vulnerabilities that were closed against one attack strategy can reopen when you swap models, update prompts, or add a new tool. Red teaming needs to be recurring, not a one-time certification.
Only testing single-turn attacks. Multi-turn strategies like Crescendo catch a meaningfully different class of failure than single-shot prompts. Skipping them leaves a real gap, especially for anything conversational.
Ignoring the `reason` field. A score of 0 without reading why is a missed diagnostic. The judge's reasoning usually points straight at the specific phrase or behavior that crossed the line, which is far more actionable than the raw pass/fail number.
Scoping vulnerabilities too broadly on every run. Running all 50+ vulnerabilities with every attack enhancement on every commit is slow and expensive. Scope your CI gate to the vulnerabilities most relevant to your domain, and expand coverage on a slower cadence.

Building This Into Your Workflow

Red teaming isn't a replacement for standard LLM evaluation — it's the other half of a complete testing strategy. Faithfulness, relevancy, and hallucination metrics tell you your system works when used as intended. Red teaming tells you what happens when it isn't. Both need to run continuously as your prompts, models, and integrations change, not just once before a demo.

Start small: pick the two or three vulnerabilities that would actually hurt your business if they surfaced in production — PII leakage for anything handling customer data, excessive agency for anything with tool access, bias for anything customer-facing — and get a scoped scan running in CI this week. Expand the vulnerability and attack coverage from there once the workflow is proven out.

If you want a structured, hands-on path through this — writing model callbacks, configuring vulnerability and attack combinations, interpreting risk assessment reports, and setting up CI gates for adversarial testing — our DeepEval Tutorial course on teachyou.ai walks through the entire red teaming workflow step by step, alongside the rest of the DeepEval evaluation ecosystem.