Generating Test Data with AI

Every test suite is only as good as the data feeding it, and AI test data generation solves the two problems that have plagued QA teams for years: data that is too clean to catch bugs, and data that takes too long to hand-craft. Instead of writing hundreds of fixture rows by hand or scrubbing production data (with all its privacy risk), you can pair a schema definition with a large language model and get realistic, varied, edge-case-aware records in seconds. This guide walks through the actual code: setting up a generator, validating output against a schema, producing adversarial edge cases, and wiring the whole thing into a CI pipeline.

What AI Test Data Generation Actually Means

AI test data generation is the practice of using a language model, alone or combined with a traditional faker library, to produce structured records that look and behave like real data without containing any real user information. There are two flavors worth separating:

Procedural generation uses libraries like Faker to produce plausible-looking names, addresses, and dates from templates and random selection. It is fast, deterministic when seeded, and completely free of any "reasoning" about the domain.

Model-driven generation uses an LLM (Claude, GPT, or a local model) to read a schema or a natural-language description and produce entries that satisfy business rules, internal consistency, and even narrative coherence, such as a support ticket whose body text actually matches its category and severity fields.

The strongest pipelines use both: Faker for the boring, high-volume scaffolding (names, emails, UUIDs) and an LLM for the parts that need judgment (realistic free-text fields, plausible-but-wrong inputs for negative testing, or domain-specific edge cases like malformed but almost-valid tax IDs).

Why Traditional Test Data Approaches Fall Short

Before writing code, it helps to be clear about what you are replacing.

• Hand-written fixtures are fast to create but tend to cluster around the "happy path." Nobody hand-writes fifty variations of a malformed phone number.
• Production data copies raise compliance and privacy concerns, and they are frozen in time, so they never reflect new fields or edge cases introduced by a recent schema change.
• Pure random generation (random strings, random integers) produces data that is technically varied but semantically meaningless. A random string in an "email" field will fail validation before your test ever reaches the logic you wanted to exercise.

AI test data generation addresses all three: it is fast to regenerate, it never touches real user records, and because the model understands field semantics, it produces variation that is meaningful rather than noise.

Setting Up Your Environment

You need three things: a schema definition library, a faker library for procedural fields, and access to an LLM API for the judgment-heavy fields.

pip install pydantic faker anthropic

Create a .env file (never commit this) with your API key:

echo "ANTHROPIC_API_KEY=your-key-here" >> .env

Define your schema first. Using Pydantic gives you validation for free later in the pipeline, which matters because generated data, whether from Faker or an LLM, still needs to be checked before it enters your test suite.

from pydantic import BaseModel, EmailStr, Field
from typing import Literal
from datetime import date

class SupportTicket(BaseModel):
    ticket_id: str = Field(pattern=r"^TCK-\d{6}$")
    customer_email: EmailStr
    subject: str = Field(max_length=120)
    body: str = Field(max_length=2000)
    category: Literal["billing", "technical", "account", "shipping"]
    priority: Literal["low", "medium", "high", "urgent"]
    created_at: date

This schema is the contract every generated record must satisfy, whether it comes from Faker, an LLM, or a hybrid of the two.

Generating the Procedural Fields with Faker

Start with the fields that do not need "understanding," just realistic variety.

from faker import Faker
import random
import uuid

fake = Faker()

def generate_base_fields():
    return {
        "ticket_id": f"TCK-{random.randint(100000, 999999)}",
        "customer_email": fake.email(),
        "created_at": fake.date_between(start_date="-1y", end_date="today"),
        "category": random.choice(["billing", "technical", "account", "shipping"]),
        "priority": random.choices(
            ["low", "medium", "high", "urgent"],
            weights=[0.4, 0.3, 0.2, 0.1],
        )[0],
    }

Notice the weighted random.choices call. Real support tickets skew toward low and medium priority, and matching that distribution matters if your tests include anything statistical, like a dashboard widget or an SLA alert threshold.

Using an LLM for the Fields That Need Judgment

The subject and body fields are where procedural generation breaks down. A Faker sentence does not know what a "billing" ticket sounds like versus a "shipping" one. This is where an LLM earns its place in the pipeline.

import anthropic
import json
import os

client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

def generate_ticket_text(category: str, priority: str, count: int = 5) -> list[dict]:
    prompt = f"""Generate {count} realistic customer support ticket subject and body
pairs for the category "{category}" with priority "{priority}".
Each body should be 2-4 sentences, written as a real customer would write it,
including occasional typos and informal tone where priority is "low" or "medium",
and more urgent, terse language for "high" or "urgent".

Return ONLY a JSON array of objects with keys "subject" and "body". No prose,
no markdown fences, just the raw JSON array."""

    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=2000,
        messages=[{"role": "user", "content": prompt}],
    )

    text = response.content[0].text.strip()
    return json.loads(text)

A few things worth calling out in that function, because they are the difference between a demo and a pipeline you can actually rely on:

• The prompt explicitly asks for raw JSON with no markdown fences. Models will happily wrap output in triple backticks unless told not to, and that wrapper breaks json.loads every time.
• Priority is fed back into the prompt so tone varies with severity, which is exactly the kind of correlation a QA reviewer would expect real data to have.
• count batches multiple records per call, which matters for cost and latency when you need thousands of rows.

Assembling Full Records

Combine the procedural and model-driven halves, then validate against the Pydantic schema before anything touches your test database.

def build_dataset(n: int = 50) -> list[SupportTicket]:
    tickets = []
    by_bucket: dict[tuple, list[dict]] = {}

    while len(tickets) < n:
        base = generate_base_fields()
        key = (base["category"], base["priority"])

        if key not in by_bucket or not by_bucket[key]:
            by_bucket[key] = generate_ticket_text(
                base["category"], base["priority"], count=5
            )

        text = by_bucket[key].pop()
        record = {**base, **text}

        try:
            tickets.append(SupportTicket(**record))
        except Exception as exc:
            print(f"Skipping invalid record: {exc}")

    return tickets

Batching text generation per (category, priority) bucket means you make far fewer API calls than generating text one record at a time, while still getting five distinct subject and body pairs to draw from before requesting more.

Generating Adversarial and Edge-Case Data

Happy-path data is only half the job. The real value of AI test data generation shows up when you ask the model to be deliberately difficult.

def generate_edge_cases() -> list[dict]:
    prompt = """Generate 15 edge-case inputs for a support ticket "customer_email"
field, designed to test input validation. Include:
- Addresses that are almost valid but technically malformed
- Unicode and internationalized domain names
- Extremely long local parts
- Addresses with plus-addressing and sub-addressing
- Common copy-paste mistakes (trailing spaces, wrapped in angle brackets)

Return ONLY a JSON array of strings."""

    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1000,
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(response.content[0].text.strip())

Feed these into a negative test rather than the main dataset:

def test_email_validation_rejects_malformed_addresses():
    edge_cases = generate_edge_cases()
    for email in edge_cases:
        record = {**generate_base_fields(), "customer_email": email,
                   "subject": "test", "body": "test"}
        record["customer_email"] = email
        try:
            SupportTicket(**record)
            print(f"WARNING: '{email}' passed validation, review manually")
        except Exception:
            pass

This pattern, asking the model to specifically target the validation logic you care about, tends to surface far more realistic failure modes than randomly mutating strings, because the model has seen the actual shapes malformed emails take in the wild.

AI Test Data Generation for API and Contract Testing

If your service exposes a JSON Schema or OpenAPI spec, you can skip hand-writing the Pydantic model and generate payloads directly from the spec description.

def generate_api_payloads(openapi_schema: dict, endpoint_path: str, count: int = 10):
    schema_snippet = json.dumps(
        openapi_schema["paths"][endpoint_path]["post"]["requestBody"],
        indent=2,
    )

    prompt = f"""Given this OpenAPI request body schema:

{schema_snippet}

Generate {count} valid request payloads that satisfy the schema, with realistic
variety in string lengths, numeric ranges, and optional field presence.
Also generate 5 additional payloads that violate exactly one constraint each,
labeled with which constraint they violate.

Return ONLY JSON with keys "valid" (array of payload objects) and
"invalid" (array of objects with "payload" and "violates" keys)."""

    response = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=3000,
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(response.content[0].text.strip())

This is particularly effective for contract testing tools like Pact or schema-validation middleware, because the "invalid" payloads come pre-labeled with exactly which rule they break, so a failing assertion tells you immediately whether your validator caught the right thing.

Validating and Sanitizing Generated Output

Never trust generated data blindly, whether it came from Faker or an LLM. Two failure modes show up repeatedly:

1. Schema drift. The model occasionally invents a field, drops a required one, or returns a type mismatch (a string where you expected an integer).
2. Leaked real-world data. LLMs trained on broad internet data can occasionally reproduce a real-looking name or company that happens to exist. For anything customer-facing, run a final filter.

import re

BLOCKLIST_PATTERNS = [
    r"@(gmail|yahoo|outlook|hotmail)\.com$",
]

def sanitize_record(record: dict) -> dict:
    email = record.get("customer_email", "")
    for pattern in BLOCKLIST_PATTERNS:
        if re.search(pattern, email, re.IGNORECASE):
            record["customer_email"] = f"testuser+{uuid.uuid4().hex[:8]}@example.test"
    return record

Routing every generated email through a fixed @example.test style domain (reserved for exactly this purpose) removes any chance of a generated address colliding with a real inbox, which matters if your test environment ever accidentally fires a real email.

Integrating AI Test Data Generation into a CI Pipeline

Generating data at test time is usually the wrong move, since it adds API latency and flakiness to every CI run. Generate once, cache the output as a fixture, and regenerate only when the schema changes.

import hashlib

def dataset_cache_key(schema_cls) -> str:
    schema_json = json.dumps(schema_cls.model_json_schema(), sort_keys=True)
    return hashlib.sha256(schema_json.encode()).hexdigest()[:12]

def get_or_generate_dataset(schema_cls, n: int, cache_dir="fixtures"):
    key = dataset_cache_key(schema_cls)
    path = f"{cache_dir}/dataset_{key}.json"

    if os.path.exists(path):
        with open(path) as f:
            return [schema_cls(**r) for r in json.load(f)]

    dataset = build_dataset(n)
    os.makedirs(cache_dir, exist_ok=True)
    with open(path, "w") as f:
        json.dump([r.model_dump(mode="json") for r in dataset], f, default=str)

    return dataset

The cache key is derived from the schema's own JSON representation, so any change to SupportTicket, a new field, a changed constraint, automatically invalidates the cache and triggers regeneration on the next test run. Commit the generated fixture file to your repository alongside the test suite, and only regenerate it in a scheduled job or a dedicated pull request, not on every CI run.

A pytest fixture wraps this cleanly:

import pytest

@pytest.fixture(scope="session")
def support_tickets():
    return get_or_generate_dataset(SupportTicket, n=100)

def test_priority_distribution_is_realistic(support_tickets):
    urgent_count = sum(1 for t in support_tickets if t.priority == "urgent")
    assert urgent_count / len(support_tickets) < 0.25

Common Pitfalls in AI Test Data Generation

• Asking for too much in one call. Requesting 500 records in a single prompt tends to produce repetitive, lower-quality output as the model runs out of "novel" variation to draw on. Batch in groups of 5-20 and loop.
• Skipping schema validation. Treat LLM output the same way you would treat user input: validate, don't trust. The Pydantic layer in this guide is not optional.
• Forgetting determinism. Model output is not reproducible run to run. If a test failure needs to be debugged later, persist the exact generated dataset (the caching pattern above) rather than regenerating on demand.
• Using production-shaped prompts without review. If your prompt describes real customer segments or actual company names to "ground" the generation, review the output for anything that looks too specific to be coincidental.
• Ignoring cost at scale. Procedural generation with Faker is essentially free; model calls are not. Reserve the LLM for the fields that actually need semantic judgment, and let Faker handle the rest.

FAQ

Is AI-generated test data safe to use instead of production data? Yes, when it is generated fresh (not derived from real records) and passed through a sanitization step like the one shown above. Unlike scrubbed production copies, properly generated synthetic data carries no re-identification risk because it was never tied to a real person to begin with.

Which fields should use Faker versus an LLM? Use Faker for anything with a fixed, well-known format: names, emails, addresses, phone numbers, UUIDs, dates. Reserve the LLM for free-text fields, fields that need to be internally consistent with other fields (a ticket body matching its category), or adversarial edge cases where you want deliberately tricky input.

How do I keep generated data consistent across test runs? Generate once, validate, and persist the result as a fixture file (JSON or a database seed script), then check it into version control. Regenerate only when the underlying schema changes, using a cache key derived from the schema itself, as shown in the caching example.

Can AI test data generation replace property-based testing tools like Hypothesis? No, they solve different problems. Property-based testing tools explore the input space systematically and shrink failing cases to a minimal reproduction. AI test data generation produces semantically realistic data. The strongest setups use both: Hypothesis for exhaustive boundary exploration, and an LLM for realistic, narrative-consistent fixtures that Hypothesis would never think to construct on its own.

How do I test that my validation logic actually rejects bad AI-generated data? Generate edge cases explicitly designed to break validation, as shown in the generate_edge_cases function, then assert that your schema or validator rejects each one. If a generated "almost valid" input passes validation unexpectedly, that is a genuine bug the AI just found for you, not a flaw in the generator.

Does model non-determinism make tests flaky? Only if you generate data at test-execution time. Generate and cache the dataset as a build artifact, and the tests that consume it run against a fixed, versioned fixture just like any hand-written one, so flakiness from the model itself never reaches your CI results.