LLM Evaluation with Braintrust: A Hands-On Guide

A braintrust llm eval is how you turn "the model feels better now" into a number you can defend in a pull request. Braintrust is a platform built specifically for testing, scoring, and comparing LLM outputs across prompt versions, models, and code changes, and it works whether you are shipping a chatbot, a RAG pipeline, or an agent that calls tools. In this guide you will set up a Braintrust project, write scorers that catch real regressions, log production traffic for continuous evaluation, and wire the whole thing into CI so a bad prompt change never reaches production unnoticed.

Why You Need a Dedicated Eval Tool

Most teams start LLM evaluation with a spreadsheet: a handful of example inputs, expected outputs pasted in by hand, and someone eyeballing the model's response each time a prompt changes. That works for a week. Then the prompt grows to 40 lines, the model gets swapped, and nobody can tell whether the new version is actually better or just different. A braintrust llm eval setup replaces that spreadsheet with code: a dataset, a task function that produces outputs, and one or more scorers that grade those outputs automatically. Every run is stored as an experiment, so you can diff two prompt versions side by side and see exactly which test cases regressed.

The three building blocks in Braintrust are:

• Dataset: the set of inputs (and optionally expected outputs) you evaluate against.
• Task: the function under test, usually a call to your LLM pipeline.
• Scorer: a function that compares the task's output to the expected output (or checks the output on its own merits) and returns a score between 0 and 1.

Once those three pieces exist, Braintrust runs every input through the task, scores every output, and gives you a dashboard showing average score, per-case scores, and diffs against previous runs.

Setting Up Your First Braintrust Eval

Install the SDK for your language. The examples below use TypeScript, but the Python SDK mirrors the same API almost one to one.

npm install braintrust autoevals

Sign up at Braintrust, create an organization, and grab an API key from your account settings. Export it as an environment variable so the SDK can authenticate:

export BRAINTRUST_API_KEY=your-key-here

Now write your first eval file. This example tests a simple summarization prompt:

import { Eval } from "braintrust";
import { Factuality } from "autoevals";

Eval("summarizer-eval", {
  data: () => [
    {
      input: "The quarterly report showed revenue up 12% but costs up 18%.",
      expected: "Revenue grew 12% while costs grew faster at 18%.",
    },
    {
      input: "The team shipped the new onboarding flow a week ahead of schedule.",
      expected: "Onboarding flow shipped one week early.",
    },
  ],
  task: async (input) => {
    const response = await callYourSummarizer(input);
    return response;
  },
  scores: [Factuality],
});

Run it with the Braintrust CLI:

npx braintrust eval summarizer-eval.ts

The CLI prints a summary table and a link to the run in the Braintrust UI. Open that link and you get a per-row breakdown: input, expected output, actual output, and the score each scorer assigned. This is the loop you will repeat dozens of times as you iterate on a prompt: change the prompt, rerun the eval, compare the new experiment against the last one.

Writing Scorers That Catch Real Regressions

The autoevals package ships pre-built scorers for common cases: Factuality checks whether a response is factually consistent with a reference answer, Levenshtein measures string similarity, AnswerCorrectness and AnswerRelevancy are built for RAG answers, and Sql validates generated SQL against expected results. These get you started fast, but generic scorers miss domain-specific failure modes. A support bot that hallucinates a refund policy will still score fine on Levenshtein if the wording is close.

Write a custom scorer as a plain function that takes input, output, and expected, and returns a number or an object with a name and score:

function containsNoPII(args) {
  const { output } = args;
  const emailPattern = /[\w.+-]+@[\w-]+\.[a-z]{2,}/i;
  const phonePattern = /\b\d{3}[-.\s]?\d{3}[-.\s]?\d{4}\b/;
  const leaked = emailPattern.test(output) || phonePattern.test(output);
  return {
    name: "no_pii_leak",
    score: leaked ? 0 : 1,
  };
}

For anything that needs judgment rather than pattern matching, use an LLM-as-judge scorer. Braintrust makes this easy with the LLMClassifierFromTemplate helper, which lets you write the grading rubric as a prompt:

import { LLMClassifierFromTemplate } from "autoevals";

const ToneCheck = LLMClassifierFromTemplate({
  name: "professional_tone",
  promptTemplate:
    "You are grading a customer support reply.\n" +
    "Reply: {{output}}\n\n" +
    "Is the tone professional and free of sarcasm? Answer Y or N.",
  choiceScores: { Y: 1, N: 0 },
  useCoT: true,
});

Pass useCoT: true and the judge model writes its reasoning before picking a label, which noticeably improves consistency on subjective checks like tone or helpfulness. Add this scorer to the scores array alongside your deterministic checks. A good braintrust llm eval usually mixes two or three scorer types: one deterministic check for hard constraints (no PII, valid JSON, correct SQL), one similarity or factuality check against a reference, and one LLM judge for qualitative traits that are hard to encode as rules.

Keep scorers narrow. A single scorer that tries to grade correctness, tone, and format compliance all at once produces a blended score that is hard to debug. When a case fails, a name like professional_tone: 0 tells you exactly what broke; a generic overall_quality: 0.4 does not.

Building Datasets That Actually Exercise Your Prompt

An eval is only as good as its dataset. Three sources cover most needs:

1. Hand-written edge cases. Deliberately adversarial inputs: empty strings, very long inputs, inputs in a different language, inputs that try to get the model to break format.
2. Real production traces. Pull a sample of real user inputs (see the logging section below) and add expected outputs, either written by a human reviewer or accepted as-is if the current output is already correct.
3. Synthetic generation. Ask an LLM to generate plausible inputs for your domain, then have a human spot-check them before adding to the dataset.

Store datasets in Braintrust directly instead of hardcoding arrays in your eval file, so non-engineers on your team can add cases through the UI without touching code:

import { initDataset } from "braintrust";

const dataset = initDataset("my-project", { dataset: "support-replies" });

for (const row of dataset) {
  // row.input, row.expected, row.metadata
}

Use the data field in Eval() to point at this dataset instead of an inline array, and every new row anyone adds through the UI automatically flows into the next eval run.

Logging Production Traces for Continuous Eval

Offline evals catch regressions before you ship, but they only cover the cases you thought to write. Braintrust also supports logging live traffic, which turns your production system into a continuously growing eval dataset. Wrap your LLM calls with the wrapOpenAI or wrapAnthropic helper (or the generic traced function for any provider) and every request gets logged automatically:

import { wrapAnthropic } from "braintrust";
import Anthropic from "@anthropic-ai/sdk";

const client = wrapAnthropic(new Anthropic());

async function generateReply(userMessage) {
  const response = await client.messages.create({
    model: "claude-sonnet-4-5",
    max_tokens: 500,
    messages: [{ role: "user", content: userMessage }],
  });
  return response.content[0].text;
}

Each call shows up in the Braintrust logs view with full input, output, latency, and token usage. From there you can:

• Tag interesting or problematic logs and promote them straight into a dataset for the next offline eval.
• Attach online scorers that run automatically on a sample of live traffic, so you get a rolling quality signal without waiting for the next eval run.
• Trace multi-step agent calls, since traced supports nested spans, so a tool call, a retrieval step, and a final generation all show up as one connected trace instead of three disconnected log lines.

For multi-step pipelines, wrap each stage in its own span so you can see exactly where time and quality are lost:

import { traced } from "braintrust";

async function ragPipeline(query) {
  const docs = await traced(
    async (span) => {
      const results = await retrieve(query);
      span.log({ input: query, output: results });
      return results;
    },
    { name: "retrieval" }
  );

  const answer = await traced(
    async (span) => {
      const response = await generateAnswer(query, docs);
      span.log({ input: { query, docs }, output: response });
      return response;
    },
    { name: "generation" }
  );

  return answer;
}

When a user reports a bad answer, you open the trace and see whether retrieval pulled the wrong documents or generation ignored good ones. That distinction is the difference between fixing a retriever and rewriting a prompt for no reason.

Comparing Experiments and Catching Regressions

Every time you run Eval(), Braintrust creates a new experiment tied to your project. The UI's comparison view lines up two experiments side by side, row by row, and highlights which scores went up, which went down, and which cases flipped from passing to failing. This is the step teams skip when they move too fast, and it is the one that actually prevents shipping a worse prompt.

A useful habit: name experiments after what changed, not after a date.

Eval("summarizer-eval", {
  experimentName: "gpt-to-claude-swap",
  data: () => dataset,
  task: myTask,
  scores: [Factuality, containsNoPII],
});

When you open the comparison view later, "gpt-to-claude-swap vs baseline" tells you immediately what the diff represents, instead of forcing you to remember what happened on a given Tuesday.

Set a score threshold as a merge gate rather than eyeballing the dashboard every time. Braintrust's CLI returns a non-zero exit code when you pass --fail-on-regression, which is exactly what a CI step needs.

Running Evals in CI

Add an eval step to your pipeline so prompt or model changes cannot merge without passing the same bar as code changes. A minimal GitHub Actions job:

name: llm-eval

on:
  pull_request:
    paths:
      - "prompts/**"
      - "src/llm/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx braintrust eval summarizer-eval.ts
        env:
          BRAINTRUST_API_KEY: ${{ secrets.BRAINTRUST_API_KEY }}

Scope the paths filter so the eval only runs when prompt files or LLM integration code actually change, otherwise every unrelated pull request pays the latency and API cost of a full eval run. For teams with several eval suites (one per feature or per prompt), run them in a matrix job so failures are attributed to the specific pipeline that broke, not a single monolithic eval script covering everything.

Treat the eval score threshold the same way you treat a test coverage threshold: strict enough to block real regressions, loose enough that it does not flake on every rerun because an LLM judge scorer has some natural variance. In practice, a threshold with a small tolerance band (say, allow the average score to drop by no more than a couple of percentage points versus baseline) works better than a hard floor, because it accounts for run-to-run noise in LLM-as-judge scores while still catching genuine regressions.

Evaluating RAG and Agent Pipelines Specifically

RAG systems need two separate evals, not one: retrieval quality and generation quality. Score retrieval by checking whether the retrieved chunks actually contain the answer (a recall-style check), independent of what the generation step does with them:

function retrievalRecall(args) {
  const { output, expected } = args;
  const retrievedText = output.map((d) => d.content).join(" ");
  const found = expected.requiredFacts.every((fact) =>
    retrievedText.includes(fact)
  );
  return { name: "retrieval_recall", score: found ? 1 : 0 };
}

Then score generation separately, assuming retrieval already succeeded, using AnswerCorrectness or a custom LLM judge that compares the final answer to a reference. Splitting the two evals means a regression in retrieval doesn't get masked by a generation step that happens to compensate, and vice versa.

Agent evals need a third dimension: tool-call correctness. Check whether the agent called the right tool with the right arguments, independent of whether the final answer was correct, since an agent can accidentally reach the right answer through the wrong tool sequence:

function correctToolCall(args) {
  const { output, expected } = args;
  const calledRight =
    output.toolName === expected.toolName &&
    JSON.stringify(output.toolArgs) === JSON.stringify(expected.toolArgs);
  return { name: "tool_call_correct", score: calledRight ? 1 : 0 };
}

Log the full tool-call trace with traced spans as shown earlier, so when this scorer fails you can immediately see which step in the agent's reasoning picked the wrong tool.

Common Pitfalls When Running a Braintrust LLM Eval

A few mistakes show up repeatedly on teams adopting Braintrust:

• Datasets that never grow. Ten hand-written examples catch obvious breakage but miss the long tail. Feed real production logs back into the dataset regularly.
• Only using LLM judges. They're flexible but noisy and cost money on every run. Pair them with deterministic checks (format validation, regex, length limits) wherever a rule-based check is possible.
• No baseline experiment. Without a fixed baseline to diff against, "the score is 0.82" tells you nothing. Always compare against the last known-good experiment, not just an absolute number.
• Ignoring latency and cost in the eval. A prompt that scores higher but doubles token usage or triples latency might not be a net win. Log metadata fields for latency and token count on every task run so they show up next to the quality score.
• Testing only the happy path. Add deliberately malformed inputs, empty inputs, and adversarial prompts to the dataset, not just realistic user queries.

FAQ

What is a braintrust llm eval used for? It is used to systematically measure whether an LLM pipeline's outputs are correct, safe, and consistent across prompt changes, model swaps, or code refactors, replacing manual spot-checking with repeatable, scored test runs.

Do I need Braintrust specifically, or can I build this myself? You can build a homegrown eval harness with a dataset loop and scoring functions, but Braintrust adds experiment comparison, production log capture, trace visualization, and a CLI that plugs into CI, which otherwise takes real engineering time to replicate.

How is this different from unit testing my prompt? A unit test usually asserts exact string equality, which breaks constantly against non-deterministic LLM output. Braintrust scorers grade on a spectrum (similarity, factuality, rule compliance) and store every run as a comparable experiment instead of a pass or fail assertion.

Can I evaluate multiple models in the same eval? Yes. Parameterize the task function to accept a model name, run the same Eval() call once per model, and compare the resulting experiments in the UI to see which model wins on which scorer and which cases.

How many scorers should one eval have? Two to four narrow scorers usually beats one broad scorer. Mix at least one deterministic check with one factuality or similarity check, and add an LLM judge only for traits that genuinely require judgment.

Does logging production traffic slow down my application? The wrapOpenAI and wrapAnthropic helpers log asynchronously after the response is returned to the caller, so they add negligible latency to the user-facing request.

How do I decide what counts as a regression in CI? Compare the new experiment's average score per scorer against your last merged baseline, and fail the build if any individual scorer drops beyond a small tolerance band, rather than relying on a single blended score.

Can non-engineers contribute to the eval dataset? Yes. Once a dataset is created with initDataset, team members can add, edit, or tag rows directly in the Braintrust UI, and those rows flow into the next eval run without any code change.