Eval-Driven Development: Writing Evals Before You Write Prompts

The prompt that worked in the demo and broke in production

Every AI engineer has lived this story. You write a prompt, run it three times in a notebook, watch it produce something impressive, and ship it. Two weeks later a support ticket lands on your desk: the model summarized a refund request as a compliment, or it hallucinated a policy that doesn't exist, or it quietly stopped following the output format after a system update changed something upstream. You go back to the prompt, tweak a sentence, run it again by hand, and convince yourself it's fixed. Nobody wrote down what "fixed" means. Nobody can tell you, six weeks from now, whether the next tweak made things better or worse.

This is the default workflow for most people building with large language models, and it is the reason so many LLM features stall out in "pretty good in the demo" purgatory. The fix isn't a smarter model or a longer prompt. It's a discipline borrowed from software engineering that most prompt writers skip entirely: write the test before you write the implementation.

That's eval-driven development. Instead of treating evaluation as the thing you do after a feature feels done, you treat it as the thing that defines what "done" means before you write a single line of prompt. You write evals first, watch them fail against a naive prompt, and then iterate until they pass — the same rhythm as test-driven development, just aimed at a system whose outputs are fuzzy, probabilistic, and stubbornly hard to unit-test with assert x == y.

This article is about why that ordering matters, what an eval actually looks like for an LLM system, and how to build the habit into your own workflow — whether you're building a customer support bot, a coding assistant, or a RAG pipeline for internal documents.

Why "just try it and see" stops working

Traditional software has a clean contract: given this input, produce that exact output. You write a test, you run the function, you get a boolean. LLM systems break that contract in three specific ways, and each one is why ad hoc prompting eventually collapses under its own weight.

Outputs are non-deterministic. The same prompt against the same model can produce different phrasing, different structure, occasionally a different answer entirely, especially at temperatures above zero. "It worked when I ran it" tells you almost nothing about whether it will work the next ninety-nine times.

There is no single correct answer. A good summary of a support ticket isn't one exact string — it's a fuzzy region of acceptable outputs that all preserve certain facts, tone, and structure. You can't diff your way to correctness.

Prompts have blast radius. Change a sentence to fix hallucination on one type of query, and you can silently break formatting on a completely different type of query you weren't even thinking about when you made the edit. Without a suite of test cases running across the whole distribution of real inputs, you have no way of knowing you just broke something three categories away.

Manual spot-checking catches none of this reliably. You test the five examples you remember to test, ship, and the sixth category — the one you didn't think of — is the one a user hits in production. Eval-driven development exists to replace "I tried it and it looked right" with a number you can track over time.

What an eval actually is

Strip away the tooling and jargon, and an eval is three things: a set of representative inputs, a way to score the output for each input, and a threshold that tells you whether the system passes. That's it. You do not need a fancy framework to start. You need a spreadsheet of test cases and the discipline to run them every time you touch the prompt.

Concretely, for a customer-support triage system, an eval dataset might look like this:

eval_cases = [
    {
        "id": "refund_request_angry",
        "input": "This is the THIRD time my order has shown up broken. I want my money back NOW.",
        "expected_category": "refund_request",
        "expected_tone": "empathetic",
        "must_not_contain": ["policy does not allow", "unfortunately we cannot"],
    },
    {
        "id": "shipping_status_neutral",
        "input": "Hey, any update on order #48213? It's been 5 days.",
        "expected_category": "shipping_status",
        "expected_tone": "neutral",
        "must_contain_placeholder": "{tracking_link}",
    },
    {
        "id": "ambiguous_multi_intent",
        "input": "The item arrived late AND it's the wrong color, also can you cancel my subscription",
        "expected_category": "multi_intent",
        "expected_behavior": "flag_for_human_review",
    },
]

Each case encodes a judgment call you already made in your head but never wrote down. The angry refund case is there because you know, from experience, that models under-index on empathy when a customer is heated and over-index on citing policy. The multi-intent case is there because you know real users don't ask one clean question at a time — they dump three problems into one message, and your first prompt draft will almost certainly only handle the first one.

Writing these cases down before you write the prompt forces you to be explicit about what "good" means. That explicitness is the entire value of the exercise. Most prompt failures aren't model failures — they're the failure of the engineer to decide, in advance, what correct behavior looks like across the full range of things users actually send.

Writing evals before prompts: the actual workflow

Here's the sequence, in practice, for a new LLM feature.

1. Collect or write 20-50 realistic input examples. Pull them from support logs, from user research, from your own imagination of edge cases. Skew toward the ugly ones — the angry customer, the ambiguous request, the message in broken English, the prompt injection attempt. The easy cases will pass with almost any prompt; the eval suite earns its keep on the hard ones.
2. For each example, write down the acceptance criteria, not the exact expected output. You're not writing expected_output = "Sorry to hear that..." — you're writing the properties a correct output must have: category correctness, tone, presence or absence of certain phrases, adherence to a JSON schema, whether it escalates to a human.
3. Write the scoring function before the prompt. This might be a simple assertion (does the JSON parse and match a schema), a substring check, a regex, or a call to another LLM acting as a judge (more on that below).
4. Run the eval suite against a deliberately naive, minimal prompt. It should fail on most cases. That's the point — you now have a baseline and a red status, exactly like a failing test in TDD.
5. Iterate on the prompt until the suite goes green, adding new cases the moment you notice a new failure mode in the wild.
6. Re-run the full suite on every prompt change, every model swap, and every system-prompt refactor, not just the cases you think are relevant to your change.

Step 4 is the one people skip, and it's the one that matters most. If you write the prompt first and the eval second, you unconsciously write eval cases that your prompt already passes — you're grading your own homework with the answer key in front of you. Writing the eval blind, before the prompt exists, is what keeps you honest.

A worked example: a JSON-extraction prompt

Let's make this concrete with a task that shows up constantly in production systems: extracting structured data from unstructured text, in this case pulling meeting details out of a free-form email.

Before touching a prompt, here's the eval harness:

import json
from dataclasses import dataclass

@dataclass
class EvalCase:
    input_text: str
    expected_fields: dict
    id: str

cases = [
    EvalCase(
        id="simple_meeting",
        input_text="Let's meet Tuesday at 3pm to discuss Q3 budget.",
        expected_fields={"day": "Tuesday", "time": "15:00", "topic": "Q3 budget"},
    ),
    EvalCase(
        id="no_time_given",
        input_text="Can we grab coffee sometime next week?",
        expected_fields={"day": None, "time": None, "topic": "coffee"},
    ),
    EvalCase(
        id="relative_date",
        input_text="Following up on yesterday — same time tomorrow works for me.",
        expected_fields={"day": "relative_unresolvable", "time": None, "topic": None},
    ),
]

def score(output_json: str, case: EvalCase) -> dict:
    try:
        parsed = json.loads(output_json)
    except json.JSONDecodeError:
        return {"id": case.id, "pass": False, "reason": "invalid_json"}

    mismatches = [
        field for field, expected in case.expected_fields.items()
        if parsed.get(field) != expected
    ]
    return {
        "id": case.id,
        "pass": len(mismatches) == 0,
        "mismatches": mismatches,
    }

def run_eval(extract_fn):
    results = [score(extract_fn(c.input_text), c) for c in cases]
    passed = sum(r["pass"] for r in results)
    print(f"{passed}/{len(results)} passed")
    for r in results:
        if not r["pass"]:
            print(f"  FAIL {r['id']}: {r.get('mismatches', r.get('reason'))}")
    return results

Notice the relative_date case. It exists precisely because someone building this feature naively will forget that "tomorrow" is meaningless without knowing when the email was sent, and the model will happily hallucinate a specific date rather than admitting it can't resolve one. That's the failure mode you want to catch in the eval suite, not in a customer complaint. It only gets written down because you sat and thought about the shape of failure *before* you started prompting — which is the entire argument for eval-driven development in one example.

Run this against a first-draft prompt, watch relative_date fail (it usually will, spectacularly, with the model inventing a calendar date), and now you have a concrete, reproducible target to fix rather than a vague sense that "the extraction is a bit unreliable sometimes."

Deterministic checks vs. LLM-as-a-Judge

Not every quality dimension can be captured with a regex or a JSON schema check. "Does this rejection email sound empathetic rather than robotic?" isn't something assert can answer. This is where you need a second model to grade the first model's output — an approach commonly called LLM-as-a-Judge.

The pattern is straightforward: you send the judge model the original input, the output being evaluated, and a rubric, and ask it to score against specific criteria rather than give a vague thumbs up or down.

JUDGE_PROMPT = """
You are grading a customer support response for quality.

Customer message: {input}
Support response: {output}

Score the response from 1-5 on each dimension. Respond only with JSON.

- empathy: Does it acknowledge the customer's frustration or urgency?
- accuracy: Does it avoid making claims about policy that weren't provided?
- actionability: Does it give the customer a clear next step?

Output format:
{{"empathy": <1-5>, "accuracy": <1-5>, "actionability": <1-5>, "reasoning": "<one sentence>"}}
"""

def judge_response(input_text, output_text, judge_model_call):
    prompt = JUDGE_PROMPT.format(input=input_text, output=output_text)
    raw = judge_model_call(prompt)
    return json.loads(raw)

A few hard-earned rules make LLM-as-a-Judge trustworthy instead of decorative:

• Use a rubric, not a vibe check. "Rate this 1-10" produces noisy, inconsistent scores. Breaking it into named dimensions with concrete definitions produces scores you can actually trust and track over time.
• Ask for reasoning alongside the score. A judge that has to justify its number in one sentence is less likely to just pattern-match on length or tone.
• Validate the judge against human labels periodically. Take twenty judged outputs, have a human independently rate them, and check agreement. If the judge and human diverge, your rubric is unclear or the judge model is a poor fit for the task — fix that before trusting it at scale.
• Use a different (often stronger) model as the judge than the one being evaluated, where possible. It's not necessarily fatal to use the same model, but a model grading its own output has a tendency to be generous with itself.
• Reserve the judge for what deterministic checks can't cover. Tone, helpfulness, and coherence are judge territory. Valid JSON, correct field extraction, and forbidden phrases are cheap deterministic checks — don't burn an API call on an LLM judge to check whether a bracket is closed.

Building the eval suite into your actual workflow

An eval suite that lives in a notebook you run manually before a big demo is better than nothing, but it decays fast. The teams that get real value from this treat the eval suite the way they treat a CI test suite: versioned, automated, and run on every change.

A few practices that make the difference:

• Store eval cases as data, not code. A JSON or YAML file of cases is easier to grow, review in a pull request, and hand off to a non-engineer (a support lead, a domain expert) who can add cases without touching your prompt logic.
• Run the full suite on every prompt change, not a subset. The whole reason blast-radius failures happen is that a fix for category A silently breaks category C, and you only notice if C is in the suite you actually ran.
• Track pass rate over time, not just pass/fail today. A dashboard that shows "82% -> 79% -> 91%" across three prompt versions tells you a story that a single green checkmark doesn't.
• Add a case the moment you find a bug in production. Every real failure is a free eval case. If a user's message broke your prompt, that exact message (redacted if needed) belongs in the suite permanently, so it can never silently regress again.
• Separate your eval set from your few-shot examples. If the same three examples you used to write the prompt are also the examples you use to grade it, you've built a system that's really good at memorizing three examples. Keep a held-out set the prompt has never seen.
• Budget for both cheap and expensive checks. Run deterministic checks (schema validation, keyword checks, length limits) on every single request in production as a lightweight guardrail. Reserve the heavier LLM-as-a-Judge pass for your offline eval suite and for periodic production sampling, since judging every live request with a second model call doubles your latency and cost.

Common objections, and why they don't hold up

"This is too much overhead for a simple prompt." It's true that a one-off script that summarizes a single document for personal use doesn't need a formal eval harness. But the moment a prompt is going to run against inputs you don't control — real users, real documents, real edge cases — the overhead of writing ten eval cases up front is trivial compared to the overhead of debugging a silent regression in production three weeks later with no baseline to compare against.

"I don't have production data yet to build eval cases from." You don't need production data to start. You need domain knowledge and imagination. Write down the ten weirdest, most adversarial, most ambiguous inputs you can think of. Add real cases as they arrive. The eval suite is meant to grow continuously, not spring into existence fully formed.

"The model changes and my evals become outdated." This is a feature, not a bug. When you swap GPT-4 for a newer model, or Claude for a different provider, rerunning the exact same eval suite is precisely how you find out whether the swap actually improved things for your use case, instead of trusting a generic benchmark that has nothing to do with your product.

"My team will never agree on a rubric." This objection usually means the team hasn't tried yet, not that agreement is impossible. In practice, getting three people to agree on "does this response sound empathetic" is far easier than it sounds once you break the vague question into two or three concrete sub-questions: does it acknowledge the customer's specific complaint, does it avoid corporate boilerplate phrases, does it offer a next step. Disagreement almost always turns out to be disagreement about which sub-question mattered, not disagreement about the underlying quality. Writing the rubric down is what surfaces that distinction, and once it's surfaced, it's usually easy to resolve in a five-minute conversation instead of a recurring argument every time someone reviews output by eye.

Who owns the eval suite

One question that trips up teams adopting this practice: whose job is it to write and maintain the evals? The instinct is to hand it to whoever wrote the prompt, but that repeats the same blind spot as writing the eval after the prompt — the person closest to the implementation is the worst-positioned to think adversarially about it. A better split, even on a small team, is to have the person who understands the domain (the support lead, the person who reads user complaints, the subject-matter expert) own the eval cases and acceptance criteria, while the person writing the prompt owns making the suite pass. This mirrors how a QA function works in traditional software: the person building the feature should not be the sole author of the definition of "correct." If you're a team of one, you can still get most of this benefit by deliberately switching hats — spend twenty minutes purely as a skeptical reviewer imagining how this will be misused or misunderstood, write those cases down, and only then switch back to being the prompt author trying to satisfy them.

This division also changes how bugs get triaged. When a production failure shows up, the reflex is to patch the prompt immediately. Under eval-driven development, the reflex should instead be: add the failing case to the suite first, confirm it fails for a reason you understand, then patch the prompt and confirm the suite goes green without turning any other case red. It's a small reordering, but skipping the "add it to the suite first" step is exactly how the same bug quietly reappears two months later when someone rewrites the prompt for an unrelated reason.

Getting started this week

You don't need an eval platform to begin. Pick one LLM feature you already have in production or are about to ship. Write fifteen input examples that represent the real range of what it will see, weighted toward the ugly cases. For each one, write down — in plain language first, then as code — what a correct output must contain, must not contain, and what format it must take. Run those fifteen cases against your current prompt and count how many actually pass. You will very likely be surprised, and that surprise is the whole point: it means you were shipping on vibes, and now you have a number.

From there, the loop is simple and it's the same loop that has made software engineering reliable for decades: red, green, refactor. Write the failing eval. Change the prompt until it passes. Add the next case the moment reality hands you one. Layer in LLM-as-a-Judge for the qualities that can't be checked with a regex, validate that judge against your own human judgment periodically, and let the pass rate — not your gut feeling after three manual test runs — be the thing that tells you when a prompt is actually ready to ship.