Ragas

Building a Ragas Dashboard for Ongoing RAG Quality Tracking

Ira Menon · May 8, 2026 · 14 min read

Why a single Ragas run is not enough

Most teams discover Ragas the same way. Someone builds a retrieval-augmented generation pipeline, notices the answers are sometimes wrong or unhelpfully vague, and reaches for a metrics library to put a number on the problem. They run ragas.evaluate() against a test set, get back a faithfulness score of 0.81 and a context precision score of 0.74, screenshot the dataframe, drop it in a Slack channel, and move on.

That single run is useful. It is also almost worthless six weeks later.

RAG systems do not stay still. The embedding model gets swapped for a cheaper one. The chunking strategy changes from fixed-size to semantic splitting. Someone bumps the top_k from 5 to 8 because it "felt better" in a demo. The underlying document corpus grows, shrinks, or gets re-indexed. The LLM powering the generator gets upgraded from one point release to the next. Each of these changes can silently move your faithfulness, answer relevancy, or context recall numbers in either direction, and if you are only running evaluations ad hoc, you will not know until a customer complains or a demo goes wrong in front of a stakeholder.

The fix is not a smarter one-off evaluation. It is a dashboard — a persistent, queryable, versioned record of Ragas scores over time, tied to the commit, the config, and the dataset that produced them. This article walks through why that matters and how to actually build one, from the storage schema to the trend charts to the alerting logic that tells you when quality has regressed before your users do.

What "ongoing tracking" actually requires

Before touching code, it helps to be precise about what a Ragas dashboard needs to do that a notebook cell does not.

Persistence. Every evaluation run has to be written somewhere durable — not just printed to stdout or held in a pandas dataframe that dies when the kernel restarts.
Provenance. Each run needs metadata: which git commit, which prompt template version, which embedding model, which retriever config, and which evaluation dataset produced these numbers. Without provenance, a score is just a number floating in space.
Comparability. You need to compare run N against run N-1, and against a rolling baseline, not just eyeball absolute values.
Segmentation. Aggregate averages hide problems. A dashboard that only shows "faithfulness: 0.83" across the whole test set will not tell you that faithfulness collapses to 0.4 specifically on multi-hop questions about pricing tiers.
Alerting. Someone needs to know within minutes, not weeks, when a metric drops below an agreed threshold.
Cheap to run repeatedly. If evaluation costs $40 and 20 minutes every time, nobody will run it on every PR. The dashboard needs a cadence that is affordable enough to actually happen — nightly on the full set, on every PR on a small canary set.

Keep this list nearby. Every design decision below traces back to one of these six requirements.

Step 1: Standardize the evaluation run as a reproducible unit

Before you can track anything over time, you need every evaluation run to produce the same shape of output, tagged with the same metadata. Treat each run as an immutable record, not a live computation you re-derive later.

Here is a wrapper around Ragas that captures the config alongside the scores:

import json
import subprocess
import time
import uuid
from dataclasses import dataclass, asdict

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)


@dataclass
class RunMetadata:
    run_id: str
    git_sha: str
    timestamp: float
    embedding_model: str
    generator_model: str
    retriever_top_k: int
    prompt_version: str
    dataset_name: str
    dataset_version: str


def get_git_sha() -> str:
    try:
        return subprocess.check_output(
            ["git", "rev-parse", "--short", "HEAD"]
        ).decode().strip()
    except subprocess.CalledProcessError:
        return "unknown"


def run_ragas_eval(
    questions: list[str],
    answers: list[str],
    contexts: list[list[str]],
    ground_truths: list[str],
    config: dict,
) -> tuple[RunMetadata, "pd.DataFrame"]:
    dataset = Dataset.from_dict(
        {
            "question": questions,
            "answer": answers,
            "contexts": contexts,
            "ground_truth": ground_truths,
        }
    )

    result = evaluate(
        dataset,
        metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    )

    metadata = RunMetadata(
        run_id=str(uuid.uuid4()),
        git_sha=get_git_sha(),
        timestamp=time.time(),
        embedding_model=config["embedding_model"],
        generator_model=config["generator_model"],
        retriever_top_k=config["retriever_top_k"],
        prompt_version=config["prompt_version"],
        dataset_name=config["dataset_name"],
        dataset_version=config["dataset_version"],
    )

    return metadata, result.to_pandas()

Two things matter here. First, git_sha ties every score back to the exact code that produced it — no more guessing whether a regression came from the retriever change or the prompt tweak, because you can git diff between two SHAs. Second, the config dictionary is passed in explicitly rather than read from environment variables scattered across the codebase, so the run is self-describing.

The dataset itself deserves the same discipline. If your "golden set" of questions and ground truths changes between runs, your trend line is comparing apples to oranges. Pin a dataset_version (a hash of the file, or a tag in your data versioning tool) and only compare runs that share it, or clearly annotate on the dashboard when the underlying set changed.

Step 2: Pick a storage layer that supports time-series queries

A Ragas score is fundamentally a time-series data point with a lot of dimensions attached: metric name, score, question ID, run ID, timestamp, and all the metadata above. You have three realistic options.

Flat files (JSONL or Parquet) in object storage. Cheapest to start with. Works fine if your dashboard reads are batch jobs that reload everything into pandas periodically. Struggles once you want per-question drill-down across hundreds of runs, because you end up scanning a lot of files.

A relational database (Postgres). The best default for most teams. You get indexing, joins between runs and metadata, and easy integration with existing BI tools. This is what the rest of this article assumes.

A dedicated time-series or observability backend (e.g., feeding into an existing metrics stack like Prometheus/Grafana, or a vendor LLM-ops tool). Worth it once you already have that infrastructure and want RAG quality metrics to sit next to your other operational dashboards.

Here's a minimal Postgres schema that supports both aggregate trend lines and per-question drill-down:

CREATE TABLE eval_runs (
    run_id UUID PRIMARY KEY,
    git_sha TEXT NOT NULL,
    run_timestamp TIMESTAMPTZ NOT NULL,
    embedding_model TEXT NOT NULL,
    generator_model TEXT NOT NULL,
    retriever_top_k INTEGER NOT NULL,
    prompt_version TEXT NOT NULL,
    dataset_name TEXT NOT NULL,
    dataset_version TEXT NOT NULL,
    triggered_by TEXT NOT NULL DEFAULT 'nightly'
);

CREATE TABLE eval_scores (
    id BIGSERIAL PRIMARY KEY,
    run_id UUID NOT NULL REFERENCES eval_runs(run_id),
    question_id TEXT NOT NULL,
    question_tag TEXT,
    metric_name TEXT NOT NULL,
    score DOUBLE PRECISION NOT NULL
);

CREATE INDEX idx_eval_scores_run ON eval_scores(run_id);
CREATE INDEX idx_eval_scores_metric ON eval_scores(metric_name);
CREATE INDEX idx_eval_runs_timestamp ON eval_runs(run_timestamp);

The question_tag column matters more than it looks. Tag each question in your golden set with a category — multi_hop, single_fact, pricing, refund_policy, whatever taxonomy fits your domain — at dataset-creation time. This is what lets the dashboard answer "did faithfulness drop overall, or just for refund questions" instead of forcing you to re-run notebooks every time someone asks that question.

Writing a run into this schema after the Ragas call is straightforward:

import psycopg2
from psycopg2.extras import execute_values


def persist_run(conn, metadata: RunMetadata, scores_df, question_tags: dict[str, str]):
    with conn.cursor() as cur:
        cur.execute(
            """
            INSERT INTO eval_runs
                (run_id, git_sha, run_timestamp, embedding_model,
                 generator_model, retriever_top_k, prompt_version,
                 dataset_name, dataset_version, triggered_by)
            VALUES (%s, %s, to_timestamp(%s), %s, %s, %s, %s, %s, %s, %s)
            """,
            (
                metadata.run_id,
                metadata.git_sha,
                metadata.timestamp,
                metadata.embedding_model,
                metadata.generator_model,
                metadata.retriever_top_k,
                metadata.prompt_version,
                metadata.dataset_name,
                metadata.dataset_version,
                "nightly",
            ),
        )

        rows = []
        for _, row in scores_df.iterrows():
            qid = str(row["question"])[:64]
            tag = question_tags.get(qid, "untagged")
            for metric in ["faithfulness", "answer_relevancy", "context_precision", "context_recall"]:
                if metric in row and row[metric] is not None:
                    rows.append((metadata.run_id, qid, tag, metric, float(row[metric])))

        execute_values(
            cur,
            """
            INSERT INTO eval_scores (run_id, question_id, question_tag, metric_name, score)
            VALUES %s
            """,
            rows,
        )
    conn.commit()

Note the defensive if metric in row and row[metric] is not None check. Ragas metrics can return NaN when a metric cannot be computed for a given row — for example, context_precision needs at least one relevant context in the retrieved set, and if retrieval returned nothing usable, you get a null rather than a zero. Silently coercing that to zero would make your dashboard lie about how bad things are (zero looks like "confidently wrong," NaN looks like "could not even measure"). Track them separately, or you will misdiagnose retrieval failures as generation failures.

Step 3: Build the trend view

With scores landing in Postgres on every run, the dashboard itself becomes a fairly standard analytics problem: aggregate by day, group by metric, plot a line. You do not need a heavyweight BI tool to start — a Streamlit app reading directly from Postgres gets you a working internal dashboard in an afternoon.

import pandas as pd
import streamlit as st
import altair as alt
from sqlalchemy import create_engine

engine = create_engine(st.secrets["DATABASE_URL"])

st.title("RAG Quality Dashboard")

query = """
    SELECT
        r.run_timestamp,
        r.git_sha,
        r.dataset_version,
        s.metric_name,
        s.question_tag,
        AVG(s.score) AS avg_score
    FROM eval_scores s
    JOIN eval_runs r ON r.run_id = s.run_id
    WHERE r.dataset_version = %(dataset_version)s
    GROUP BY r.run_timestamp, r.git_sha, r.dataset_version, s.metric_name, s.question_tag
    ORDER BY r.run_timestamp
"""

df = pd.read_sql(query, engine, params={"dataset_version": "v3-2026-06"})

metric_choice = st.selectbox("Metric", df["metric_name"].unique())
filtered = df[df["metric_name"] == metric_choice]

chart = (
    alt.Chart(filtered)
    .mark_line(point=True)
    .encode(
        x="run_timestamp:T",
        y=alt.Y("avg_score:Q", scale=alt.Scale(domain=[0, 1])),
        color="question_tag:N",
        tooltip=["run_timestamp", "git_sha", "question_tag", "avg_score"],
    )
    .properties(height=400)
)

st.altair_chart(chart, use_container_width=True)

st.subheader("Latest run vs. 7-run rolling baseline")
latest = filtered.sort_values("run_timestamp").tail(1)
baseline = filtered.sort_values("run_timestamp").tail(8).head(7)["avg_score"].mean()
st.metric(
    label=metric_choice,
    value=round(latest["avg_score"].values[0], 3),
    delta=round(latest["avg_score"].values[0] - baseline, 3),
)

Two design choices worth calling out. First, the query pins dataset_version because comparing scores across different golden sets is meaningless — always let the viewer filter to a consistent dataset before looking at trends. Second, the "latest vs. rolling baseline" metric is more useful day-to-day than an absolute threshold, because it surfaces gradual drift (each run only slightly worse than the last) that a fixed pass/fail gate would miss for weeks.

Segment by question_tag in the color encoding rather than only showing one aggregate line. This is the single highest-leverage addition to a Ragas dashboard: it turns "quality went down" into "quality went down specifically for multi-hop questions after the retriever change on Tuesday," which is an actionable finding instead of a vague alarm.

Step 4: Wire it into CI as a quality gate, not just a report

A dashboard nobody is forced to look at will eventually be ignored. The more durable pattern is to run a cheap subset of the evaluation on every pull request that touches retrieval, prompting, or model config, and fail the build if scores regress past a tolerance.

import sys

REGRESSION_THRESHOLDS = {
    "faithfulness": 0.05,
    "answer_relevancy": 0.05,
    "context_precision": 0.08,
    "context_recall": 0.08,
}


def check_regression(current_scores: dict[str, float], baseline_scores: dict[str, float]) -> list[str]:
    failures = []
    for metric, tolerance in REGRESSION_THRESHOLDS.items():
        current = current_scores.get(metric)
        baseline = baseline_scores.get(metric)
        if current is None or baseline is None:
            continue
        drop = baseline - current
        if drop > tolerance:
            failures.append(
                f"{metric} dropped {drop:.3f} (baseline {baseline:.3f} -> current {current:.3f}), "
                f"exceeds tolerance {tolerance:.3f}"
            )
    return failures


def main():
    current = fetch_latest_run_scores()       # implement against your schema
    baseline = fetch_baseline_scores(window=14)  # e.g. median of last 14 nightly runs

    failures = check_regression(current, baseline)
    if failures:
        print("RAG quality regression detected:")
        for f in failures:
            print(f"  - {f}")
        sys.exit(1)

    print("No regression detected. All metrics within tolerance.")


if __name__ == "__main__":
    main()

Run this on a small canary set (30-50 representative questions covering your tagged categories) on every PR — it should finish in a couple of minutes and cost a few cents in LLM calls for the judge model. Reserve the full golden set (hundreds or thousands of questions) for a nightly scheduled run, which is where the dashboard's long-term trend lines actually get their data. This two-speed cadence satisfies the "cheap enough to run repeatedly" requirement from the top of this article without sacrificing depth.

Set the tolerance thresholds deliberately, not arbitrarily. Look at the natural run-to-run variance in your nightly data first — Ragas metrics rely on an LLM-as-judge, and that judge is not perfectly deterministic even at low temperature, so there will be some baseline noise. If your faithfulness score naturally wobbles by plus or minus 0.03 between two runs with zero code changes, a tolerance of 0.02 will trigger false alarms constantly and your team will start ignoring the gate. Set thresholds a comfortable margin above the observed noise floor.

Step 5: Track cost and latency alongside quality

A dashboard that only shows Ragas scores tells half the story. RAG quality does not exist in a vacuum — a retriever config that pushes context precision from 0.78 to 0.85 by fetching top_k=15 instead of top_k=5 also multiplies your token spend and latency. Decisions about acceptable trade-offs need both numbers in the same view.

Extend the schema with a lightweight cost/latency table keyed to the same run_id:

CREATE TABLE eval_run_costs (
    run_id UUID PRIMARY KEY REFERENCES eval_runs(run_id),
    total_input_tokens BIGINT NOT NULL,
    total_output_tokens BIGINT NOT NULL,
    estimated_cost_usd NUMERIC(10, 4) NOT NULL,
    avg_latency_ms INTEGER NOT NULL,
    p95_latency_ms INTEGER NOT NULL
);

Capture these numbers from the same pipeline invocation that generates the answers being scored — most LLM SDKs return token usage on every call, so accumulate it as you go rather than estimating after the fact. On the dashboard, plot cost-per-query and average faithfulness on the same time axis. It turns "why did we switch embedding models" into a defensible, visible trade-off instead of a decision made once in a meeting and then forgotten.

Step 6: Handle dataset drift honestly

One failure mode specific to ongoing tracking, as opposed to one-off evaluation, is dataset staleness. Your golden set was written against the product as it existed when you wrote it. Six months later, the product has three new features, two deprecated ones, and a pricing page that changed twice. If you never touch the golden set, your dashboard will report stable, healthy scores on questions that increasingly do not reflect what real users ask.

Two practical habits keep this in check. First, sample real production queries (with PII scrubbed) on a regular cadence and route a fraction of them into the golden set, replacing the oldest or least-representative entries. Second, whenever the golden set changes, bump dataset_version and keep the old version's history intact rather than overwriting it — this is exactly why the schema above ties every run to a specific dataset version instead of assuming one static "the" dataset. On the dashboard, annotate the timeline with vertical markers wherever the dataset version changes, so a viewer immediately understands why a trend line has a discontinuity and does not mistake a dataset change for a quality regression or improvement.

Step 7: Close the loop with root-cause drill-down

The final piece that separates a dashboard from a report is the ability to go from "this metric dropped" to "here is the specific retrieved context and generated answer that caused it" without leaving the tool. Store enough of the raw evaluation output to support this, even if it means a bit more storage.

CREATE TABLE eval_row_details (
    id BIGSERIAL PRIMARY KEY,
    run_id UUID NOT NULL REFERENCES eval_runs(run_id),
    question_id TEXT NOT NULL,
    question_text TEXT NOT NULL,
    retrieved_contexts JSONB NOT NULL,
    generated_answer TEXT NOT NULL,
    ground_truth TEXT
);

A simple Streamlit expander that lets a viewer click on any point in the trend chart and see the ten lowest-scoring questions from that run, alongside the actual context and answer, turns "faithfulness dropped 0.06" into "the retriever started pulling last year's pricing PDF instead of the current one" in about thirty seconds. That is the entire point of building this in the first place — not to produce a prettier chart, but to shorten the distance between noticing a problem and understanding it.

Bringing it together

None of the individual pieces here are exotic. A Postgres table, a scheduled job, a Streamlit page, and a CI check are all things most engineering teams already know how to build. The value is in the discipline of connecting them: every score tagged with the code and config that produced it, every dataset version tracked explicitly, every run cheap enough to happen automatically, and every regression visible within a day instead of discovered by a customer.

If you are evaluating RAG systems today with nothing more than a notebook you re-run when someone remembers to, the upgrade path described here is incremental — you can add the metadata wrapper this week, the Postgres schema next week, and the CI gate once you trust the noise floor. You do not need all seven steps on day one to get real value; even steps one and two alone will save you from the most common failure, which is not knowing whether last month's numbers still apply.

If you want to go deeper on the metrics themselves — how faithfulness, context precision, and context recall are actually computed, what their failure modes are, and how to build custom metrics for domain-specific evaluation — that is exactly what we cover hands-on in the Ragas Tutorial course on teachyou.ai, where you will build an evaluation pipeline like the one in this article from scratch.