Building an Eval Dashboard: Tracking Quality Over Time

The Day Nobody Noticed the Model Got Worse

Three weeks after a prompt tweak shipped to production, a support lead mentioned that the AI assistant had "been kind of useless lately." Nobody on the engineering team had noticed. There was no alert, no dashboard, no red line on a graph. The only signal was a human getting annoyed enough to say something out loud.

This is the default state of most LLM applications: quality drifts, nobody watches it, and the first sign of trouble is a complaint. It doesn't have to be this way. The fix isn't a smarter model or a longer prompt — it's an eval dashboard. A place where every change to your system (prompt edits, model swaps, retrieval tweaks, temperature changes) gets measured against a consistent yardstick, and where that measurement is visible over time instead of buried in a one-off notebook.

This article walks through how to actually build one: what to track, how to score outputs, how to store the results, and how to turn raw numbers into a dashboard your team will actually check before they ship. We'll write real code, not pseudocode, and we'll end with why LLM-as-a-Judge scoring is the piece that makes the whole system scale.

Why "Eyeballing It" Stops Working

Early on, most teams evaluate their LLM app the same way: someone runs a few prompts, reads the outputs, and says "looks good." This works when you have five test cases and one person making changes. It falls apart fast for a few concrete reasons.

• Non-determinism. The same prompt can produce a good answer once and a subtly wrong one the next time. A single manual check tells you almost nothing about the distribution of outcomes.
• Regression blindness. You fix one failure mode and silently break three others. Without a fixed test set scored consistently, you have no way to know.
• No historical record. Three months from now, "it felt worse after that update" is not a debuggable claim. A dashboard turns that feeling into a chart with a date on the x-axis.
• Scaling past one person's judgment. Once a team has five engineers shipping prompt changes independently, "looks good to me" from five different people means five different bars.

An eval dashboard solves this by making quality a number, not a feeling — and by making that number visible to everyone who can affect it, at every commit, not just when someone remembers to check.

What an Eval Dashboard Actually Tracks

Before writing code, decide what "quality" means for your application. Vague goals produce vague dashboards. Concretely, a useful eval dashboard tracks a handful of layers:

• Task success rate — did the model do the thing it was asked to do (extract the right field, answer the question correctly, generate valid JSON)?
• Groundedness / hallucination rate — for RAG or tool-using systems, does the output stay faithful to the retrieved context?
• Format compliance — does the output parse as valid JSON, match the required schema, stay under a token budget?
• Latency and cost — not "quality" in the traditional sense, but every serious dashboard tracks these alongside correctness, because a 40% quality gain that triples your API bill needs a human decision, not a silent merge.
• Safety and refusal behavior — does the model refuse when it shouldn't, or comply when it shouldn't?
• Regression deltas — how does this run compare to the last run on the same test set?

Each of these becomes a metric with a timestamp, a version tag (model, prompt version, commit hash), and a score. That's the entire data model. Everything else — the pretty charts, the alerts, the Slack notifications — is built on top of that simple shape.

Step 1: Build a Golden Dataset You Trust

The dashboard is only as good as the test cases feeding it. Most teams under-invest here and end up with either too few examples (statistical noise) or examples that don't represent real usage (false confidence).

A golden dataset needs three kinds of examples:

• Happy path cases — realistic, common inputs your system should nail every time.
• Edge cases — ambiguous queries, adversarial inputs, malformed data, empty strings, extremely long context.
• Known failure cases — bugs you've already fixed. Every regression you catch in production should graduate into a permanent test case so it can never silently come back.

Here's a minimal schema for a golden dataset entry:

# golden_dataset.py
from dataclasses import dataclass, field
from typing import Optional

@dataclass
class EvalCase:
    id: str
    input: str
    expected_output: Optional[str] = None  # exact match or reference answer
    expected_criteria: list[str] = field(default_factory=list)  # for rubric-based grading
    category: str = "happy_path"  # happy_path | edge_case | regression
    metadata: dict = field(default_factory=dict)

golden_set = [
    EvalCase(
        id="refund_policy_001",
        input="Can I get a refund if I bought the course 45 days ago?",
        expected_criteria=[
            "States the refund window is 30 days",
            "Politely explains the request is outside the window",
            "Does not fabricate a policy exception",
        ],
        category="happy_path",
    ),
    EvalCase(
        id="empty_input_002",
        input="",
        expected_criteria=["Asks a clarifying question instead of guessing"],
        category="edge_case",
    ),
    EvalCase(
        id="regression_003",
        input="What's the price in INR for the Bootcamp bundle?",
        expected_criteria=[
            "Returns a number in INR, not USD",
            "Does not include the word 'approximately' without a number",
        ],
        category="regression",
        metadata={"bug_ref": "JIRA-482"},
    ),
]

Start with 30-50 cases. That's enough to catch obvious regressions without becoming a maintenance burden. Grow it over time as you find new failure modes — but resist the urge to write hundreds of cases up front. A smaller set you actually run on every change beats a huge set that's too slow to run and gets skipped.

Where do the first 30 cases come from? Don't invent them from imagination — pull them from real usage. If you have any production logs, support tickets, or a beta user group, mine those transcripts for the inputs people actually send. Synthetic test cases written by an engineer tend to be cleaner and more polite than real traffic, which means they systematically under-represent the messy inputs that break systems in production: typos, mixed languages, half-finished sentences, users pasting in a wall of unrelated text before asking their actual question. A golden dataset built entirely from an engineer's imagination will pass beautifully and still miss the failure mode that shows up in week one of launch.

It also helps to tag each case with the specific capability it's testing, not just a broad category. "regression_003" above isn't just "a pricing question" — it's specifically testing currency handling, which means when you later add a new pricing tier or a new currency, you know exactly which existing cases need a sibling test rather than guessing at coverage.

Step 2: Choose Your Scoring Methods

Not every output can be scored the same way. Match the grading method to the task:

• Exact match / regex — great for structured outputs like classification labels, JSON schemas, or extracted fields. Cheap, deterministic, zero ambiguity.
• Rubric-based LLM grading — for open-ended text, use a second model call to grade the output against a checklist (this is LLM-as-a-Judge, covered in depth below).
• Human review — the gold standard for calibrating your automated graders, but too slow and expensive to run on every commit. Use it to spot-check and to validate the judge model periodically.
• Programmatic checks — code that verifies things like "does this SQL query actually run," "is this JSON valid," "is the URL reachable." These are underused and extremely cheap to add.

A pragmatic eval pipeline mixes all four, weighted by how expensive and how reliable each is. Deterministic checks run on every case. LLM-as-a-Judge fills the gap for subjective quality. Human review runs on a sample, weekly.

Step 3: Write the Eval Runner

The eval runner is the piece of code that takes your golden dataset, runs it against your current system, scores the results, and writes them somewhere durable. Here's a working example using a simple scoring pipeline:

# eval_runner.py
import json
import time
import uuid
from datetime import datetime, timezone

from golden_dataset import golden_set
from my_app import generate_response  # your actual LLM call
from judge import judge_response       # LLM-as-a-Judge scorer


def run_eval(model_version: str, prompt_version: str):
    run_id = str(uuid.uuid4())
    results = []

    for case in golden_set:
        start = time.time()
        output = generate_response(case.input, model=model_version)
        latency_ms = (time.time() - start) * 1000

        if case.expected_output:
            score = 1.0 if output.strip() == case.expected_output.strip() else 0.0
            method = "exact_match"
        else:
            score, rationale = judge_response(
                input_text=case.input,
                output_text=output,
                criteria=case.expected_criteria,
            )
            method = "llm_judge"

        results.append({
            "run_id": run_id,
            "case_id": case.id,
            "category": case.category,
            "score": score,
            "method": method,
            "latency_ms": round(latency_ms, 1),
            "model_version": model_version,
            "prompt_version": prompt_version,
            "timestamp": datetime.now(timezone.utc).isoformat(),
        })

    return results


def save_results(results: list[dict], path: str = "eval_runs.jsonl"):
    with open(path, "a") as f:
        for r in results:
            f.write(json.dumps(r) + "\n")


if __name__ == "__main__":
    results = run_eval(model_version="gpt-5-mini", prompt_version="v14")
    save_results(results)
    avg_score = sum(r["score"] for r in results) / len(results)
    print(f"Run complete. Average score: {avg_score:.2%} across {len(results)} cases.")

Notice this writes to a plain JSONL file. That's intentional — you don't need a fancy database to start. A flat append-only log is durable, diffable, and easy to load into pandas, DuckDB, or a real database later. Don't let "we need infrastructure first" block you from running your first eval today.

Step 4: Build the LLM-as-a-Judge Scorer

This is the piece that lets you grade open-ended, subjective outputs at scale without a human in the loop for every run. The trick is to give the judge model a narrow, checklist-style rubric rather than asking it to vaguely rate quality from 1-10.

# judge.py
import json
from anthropic import Anthropic

client = Anthropic()

JUDGE_PROMPT = """You are grading an AI assistant's response against a checklist.

User input: {input_text}

Assistant response: {output_text}

Checklist criteria:
{criteria_list}

For each criterion, answer true or false based ONLY on the response above.
Do not be lenient. If a criterion is ambiguous, mark it false.

Return strict JSON in this exact format:
{{"criteria_results": [{{"criterion": "...", "passed": true}}, ...], "rationale": "one sentence summary"}}
"""


def judge_response(input_text: str, output_text: str, criteria: list[str]):
    criteria_list = "\n".join(f"- {c}" for c in criteria)
    prompt = JUDGE_PROMPT.format(
        input_text=input_text,
        output_text=output_text,
        criteria_list=criteria_list,
    )

    message = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=500,
        messages=[{"role": "user", "content": prompt}],
    )

    raw = message.content[0].text
    parsed = json.loads(raw)

    passed_count = sum(1 for c in parsed["criteria_results"] if c["passed"])
    score = passed_count / len(criteria) if criteria else 0.0

    return score, parsed["rationale"]

Two design choices here matter a lot in practice. First, the judge grades a checklist of true/false criteria instead of a single subjective score — this reduces variance in the judge's own output and makes disagreements easier to debug ("which criterion did it fail?" instead of "why did it say 6/10?"). Second, use a different, typically stronger model as the judge than the one being evaluated, to reduce the risk of a model rating its own output favorably.

Always validate your judge against human ratings on a sample before trusting it fully. If the judge and a human reviewer disagree more than roughly 10-15% of the time on your checklist criteria, revise the rubric or the prompt before relying on it for dashboard numbers.

It's worth running this validation as its own small experiment before you build anything else. Take 20-30 outputs, have a human on your team score them against the checklist independently, then run the same cases through the judge and compare. When the two disagree, read the specific case rather than just noting the mismatch rate — you'll usually find a pattern, like the judge being too generous with responses that "sound confident" regardless of correctness, or too strict on answers that satisfy the spirit of a criterion using different wording than the checklist expected. Fixing that pattern in the judge prompt is a much better use of an afternoon than discovering it three months in, after dozens of ship decisions were already made on top of a biased number.

One more failure mode worth naming: judge drift. If you ever change the judge model itself — a version bump, a provider switch — treat that as a breaking change to your entire historical trend line. Scores from the old judge and the new judge are not directly comparable, even if both are well-calibrated individually, because their leniency and blind spots differ. Re-run the validation sample against the new judge, note the switch with a version tag in your stored results, and expect a visible step-change in the trend line that has nothing to do with your actual product quality.

Step 5: Store Results for Trend Analysis

A single eval run tells you today's score. The dashboard's real value is in comparing runs over time. Move from a flat file to a small database once you're running evals regularly — even SQLite is enough for most teams.

# storage.py
import sqlite3
from contextlib import contextmanager

DB_PATH = "evals.db"


@contextmanager
def get_conn():
    conn = sqlite3.connect(DB_PATH)
    try:
        yield conn
        conn.commit()
    finally:
        conn.close()


def init_db():
    with get_conn() as conn:
        conn.execute("""
            CREATE TABLE IF NOT EXISTS eval_results (
                run_id TEXT,
                case_id TEXT,
                category TEXT,
                score REAL,
                method TEXT,
                latency_ms REAL,
                model_version TEXT,
                prompt_version TEXT,
                timestamp TEXT
            )
        """)


def insert_results(results: list[dict]):
    with get_conn() as conn:
        conn.executemany(
            """INSERT INTO eval_results
               (run_id, case_id, category, score, method, latency_ms,
                model_version, prompt_version, timestamp)
               VALUES (:run_id, :case_id, :category, :score, :method,
                       :latency_ms, :model_version, :prompt_version, :timestamp)""",
            results,
        )


def get_trend(days: int = 30):
    with get_conn() as conn:
        cursor = conn.execute("""
            SELECT DATE(timestamp) as day, prompt_version,
                   AVG(score) as avg_score, COUNT(*) as n
            FROM eval_results
            WHERE timestamp >= DATE('now', ? || ' days')
            GROUP BY day, prompt_version
            ORDER BY day ASC
        """, (f"-{days}",))
        return cursor.fetchall()

With this in place, "did quality drop after last week's prompt change" becomes a single query instead of a guess.

Step 6: Visualize It — Make the Dashboard Impossible to Ignore

A dashboard nobody looks at is worthless. Keep the visualization simple and put it somewhere the team already lives — a lightweight Streamlit app, a Grafana panel, or even a scheduled Slack message beats a beautiful internal tool that requires a login nobody remembers.

A minimal Streamlit dashboard can be built in under 40 lines:

# dashboard.py
import streamlit as st
import pandas as pd
from storage import get_trend

st.title("Eval Quality Dashboard")

rows = get_trend(days=60)
df = pd.DataFrame(rows, columns=["day", "prompt_version", "avg_score", "n"])

st.subheader("Average score over time")
pivot = df.pivot(index="day", columns="prompt_version", values="avg_score")
st.line_chart(pivot)

st.subheader("Latest run breakdown")
latest_version = df.sort_values("day").iloc[-1]["prompt_version"]
latest = df[df["prompt_version"] == latest_version]
st.dataframe(latest)

if df["avg_score"].iloc[-1] < 0.85:
    st.warning("Quality dropped below 85% threshold on the most recent run.")

The threshold warning at the bottom matters more than the chart above it. A line graph is nice to look at; a red warning banner is what actually gets a Slack message sent and a PR blocked. Wire the same threshold check into CI so a pull request that drops eval scores below your bar fails the build, the same way a broken unit test would.

Step 7: Wire Evals Into Your Workflow, Not Just Your Dashboard

A dashboard that only gets checked manually will get forgotten within a month. The goal is to make evaluation a forcing function, not an optional dashboard visit.

• Run on every prompt or model change. Add the eval runner as a CI step so no prompt edit merges without a score.
• Tag every run with a version. Model name, prompt version, retrieval config — whatever changed should be traceable back to a specific eval run.
• Alert on regressions, not just log them. A Slack webhook that fires when the score drops more than a few points compared to the last run turns passive data into an active signal.
• Review weekly, not just when something breaks. Even a five-minute look at the trend line each week catches slow drift that a single sharp regression check would miss.
• Retire stale test cases. As your product changes, some golden dataset entries stop being representative. Prune and add continuously — this is a living dataset, not a one-time deliverable.

Common Mistakes Teams Make

A few patterns show up repeatedly when teams build their first eval dashboard, worth calling out directly.

• Grading with a single overall score. "Rate this response 1-10" produces noisy, inconsistent numbers. Checklist-based grading is more stable and far easier to debug when scores drop.
• Never validating the judge model. Teams trust LLM-as-a-Judge output blindly, then discover months later that the judge has a systematic bias (too lenient on long answers, too harsh on short ones) that's been quietly polluting the entire dashboard.
• Testing only happy paths. A 95% score that only covers easy cases hides the failures that actually reach users. Weight your golden dataset toward edge cases and past regressions, not just clean examples.
• Ignoring cost and latency. A prompt change that improves scores by 3% but doubles latency or triples token spend needs a deliberate tradeoff decision, not a silent merge.
• Treating the dashboard as a one-time project. The dashboard needs the same maintenance as any other piece of production infrastructure — dataset upkeep, judge recalibration, threshold tuning as the product evolves.

Closing: Turning Vibes Into a Discipline

The gap between teams that ship reliable LLM products and teams that ship things that quietly degrade usually isn't model choice or prompt cleverness — it's whether someone built the measurement system in the first place. An eval dashboard doesn't need to be sophisticated on day one. A JSONL file, a handful of golden test cases, and a scoring function you run before every merge will catch more regressions than months of manual spot-checking ever will.

The piece that makes this scale past a tiny hand-labeled test set is LLM-as-a-Judge: a second model, grading against a narrow checklist, validated periodically against human judgment. Get that loop right — golden dataset, deterministic checks where you can, judge-based grading where you can't, all logged with a version tag and visualized on a trend line — and quality stops being a feeling your support team reports three weeks late. It becomes a number on a dashboard that goes red before your users notice anything at all.