Automating Ragas Reports for Stakeholder Reviews

Why Your Ragas Numbers Never Leave Your Notebook

You ran the evaluation. Faithfulness came back at 0.81, context precision at 0.74, answer relevancy at 0.89. You stared at the dataframe, felt a small flicker of pride, and then closed the notebook. Three weeks later, when your product manager asks "how's the RAG pipeline doing this sprint," you have nothing to show except a vague memory of numbers that have since changed twice.

This is the most common failure mode for teams adopting Ragas: the evaluation itself is easy, but the *reporting* is an afterthought. Ragas gives you a dataframe of scores per test case. It does not give you a Slack message, a PDF, a trend line, or a dashboard your VP of Product can glance at during a standup. That gap between "we ran an eval" and "the business understands what the eval means" is where most AI teams quietly lose credibility with stakeholders who don't read pandas output.

The fix isn't a fancier metric. It's automation. If you treat Ragas evaluation the same way you treat CI test suites — triggered automatically, aggregated consistently, and pushed to a destination humans actually check — the reporting problem disappears. This article walks through building that pipeline end to end: structuring evaluation runs so they're reproducible, aggregating results into stakeholder-friendly summaries, scheduling the whole thing, and routing output to Slack, email, and a persistent store so you get trend lines instead of one-off snapshots.

What Ragas Actually Outputs (and Why Raw Scores Aren't a Report)

Before automating anything, it helps to be precise about what Ragas hands back. When you call evaluate(), you get an EvaluationResult object that wraps a pandas DataFrame — one row per test case, one column per metric.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall
from datasets import Dataset

eval_dataset = Dataset.from_dict({
    "question": questions,
    "answer": generated_answers,
    "contexts": retrieved_contexts,
    "ground_truth": reference_answers,
})

result = evaluate(
    eval_dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)

df = result.to_pandas()
print(df.head())

That dataframe is precise and per-example, which is exactly what you want for debugging a regression. It is exactly *not* what you want to hand a stakeholder. A director of product does not want 240 rows of per-question faithfulness scores. They want three sentences: "Faithfulness dropped 6 points after last week's retriever change. Context precision is stable. Here's the one failing category worth discussing."

That translation step — from raw per-row metrics to a digestible narrative — is the part almost nobody automates, and it's the part that actually earns trust. The automation isn't just "run this on a cron." It's "run this on a cron, aggregate it the same way every time, and flag what changed."

A report is only as good as the run behind it. If your eval dataset changes every time, or your model version isn't pinned, or you don't tag runs with metadata, you'll produce reports that can't be compared to each other — and an unbenchmarked report is worse than no report, because it creates false confidence.

Start by wrapping every run in a metadata envelope:

import json
import uuid
from datetime import datetime, timezone

def build_run_metadata(config: dict) -> dict:
    return {
        "run_id": str(uuid.uuid4()),
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "git_commit": config.get("git_commit"),
        "pipeline_version": config.get("pipeline_version"),
        "eval_dataset_version": config.get("eval_dataset_version"),
        "llm_judge_model": config.get("llm_judge_model", "gpt-4o-mini"),
        "num_test_cases": config.get("num_test_cases"),
    }

This metadata is what turns a single evaluation into a *comparable* evaluation. Without a git_commit and eval_dataset_version attached to each run, you cannot answer the single most important stakeholder question: "did the score change because the system got worse, or because the test set changed?"

Next, keep your test set under version control the same way you'd version a database migration. A golden_dataset.jsonl file with a changelog is enough for most teams — you don't need a fancy dataset registry on day one, you just need to stop silently editing the file that everything gets scored against.

Aggregating Scores Into Something a Human Can Read in 30 Seconds

Here's the core transformation function. It takes the raw Ragas dataframe and produces a compact summary object: overall averages, per-metric deltas versus the previous run, and the worst-performing examples worth a human's attention.

import pandas as pd

METRIC_THRESHOLDS = {
    "faithfulness": 0.80,
    "answer_relevancy": 0.85,
    "context_precision": 0.70,
    "context_recall": 0.70,
}

def summarize_run(df: pd.DataFrame, previous_summary: dict | None = None) -> dict:
    metric_cols = [c for c in df.columns if c in METRIC_THRESHOLDS]

    averages = {m: round(df[m].mean(), 3) for m in metric_cols}
    failing_rate = {
        m: round((df[m] < METRIC_THRESHOLDS[m]).mean(), 3)
        for m in metric_cols
    }

    deltas = {}
    if previous_summary:
        for m in metric_cols:
            prev = previous_summary["averages"].get(m)
            deltas[m] = round(averages[m] - prev, 3) if prev is not None else None

    worst_cases = (
        df.assign(composite=df[metric_cols].mean(axis=1))
        .nsmallest(5, "composite")[["question", "composite"] + metric_cols]
        .to_dict(orient="records")
    )

    return {
        "averages": averages,
        "failing_rate": failing_rate,
        "deltas": deltas,
        "worst_cases": worst_cases,
        "sample_size": len(df),
    }

Two design choices here matter more than they look. First, failing_rate — the percentage of test cases below a threshold — is often more actionable for stakeholders than a raw average. An average faithfulness of 0.81 sounds fine. "12% of answers failed the faithfulness bar" sounds like something to fix this sprint. Second, worst_cases gives engineers a direct pointer into what to debug next, so the report isn't just a scorecard, it's a work queue.

Set your thresholds deliberately, not arbitrarily. Pull them from an initial baseline run plus whatever your product tolerance actually is — a medical-information assistant should have a much higher faithfulness bar than an internal FAQ bot.

It's also worth deciding upfront whether you want a single composite health score or a metric-by-metric view in the headline of your report. Composite scores are tempting because they compress everything into one number a VP can track quarter over quarter, but they hide which underlying metric is driving the movement. A composite score can stay flat while faithfulness drops and answer relevancy rises to compensate — which looks fine on a chart and is actually a serious regression if faithfulness is the metric that matters most for your use case (say, a legal or medical assistant). A reasonable middle ground is to publish the composite as the headline number for executives, but always attach the per-metric breakdown directly beneath it so anyone who wants to click one level deeper can see what's actually moving.

def composite_score(averages: dict, weights: dict | None = None) -> float:
    weights = weights or {m: 1.0 for m in averages}
    total_weight = sum(weights.get(m, 0) for m in averages)
    weighted_sum = sum(averages[m] * weights.get(m, 0) for m in averages)
    return round(weighted_sum / total_weight, 3) if total_weight else 0.0

Weighting lets you encode product priorities directly into the report. A customer support bot might weight faithfulness and context precision heavily and care less about answer relevancy phrasing; a creative-writing assistant might invert that. Whatever weights you choose, write them down next to the report template itself, not buried in a config file nobody reviews — stakeholders will eventually ask "why does faithfulness count double," and you want a one-line answer ready.

Turning the Summary Into a Stakeholder-Readable Report

With a summary object in hand, rendering it as Markdown (which you can pipe into Slack, email, or a wiki page) is mechanical:

def render_markdown_report(summary: dict, run_meta: dict) -> str:
    lines = [
        f"# Ragas Evaluation Report — {run_meta['timestamp'][:10]}",
        f"Run ID: `{run_meta['run_id'][:8]}` | Commit: `{run_meta.get('git_commit', 'n/a')[:7]}` "
        f"| Sample size: {summary['sample_size']}",
        "",
        "## Summary",
    ]

    for metric, avg in summary["averages"].items():
        delta = summary["deltas"].get(metric)
        arrow = ""
        if delta is not None:
            arrow = f" ({'+' if delta >= 0 else ''}{delta} vs last run)"
        fail_pct = summary["failing_rate"][metric] * 100
        lines.append(f"- **{metric}**: {avg}{arrow} — {fail_pct:.0f}% below threshold")

    lines.append("")
    lines.append("## Worst Performing Cases")
    for i, case in enumerate(summary["worst_cases"], start=1):
        lines.append(f"{i}. \"{case['question'][:80]}...\" — composite score {case['composite']:.2f}")

    return "\n".join(lines)

The output reads like a short status update, not a data dump — because that's what stakeholders actually consume. Anyone who wants the full detail can click through to the underlying dataframe (stored separately, see the persistence section below), but the default view is three bullet points and a top-five list.

One nuance worth calling out: don't hide regressions inside a wall of green checkmarks. If context recall dropped 8 points, that line should visually stand out — bold it, or move it above the fold — rather than sitting alphabetically between two metrics that are fine. Stakeholder reports fail when everything looks equally important.

Scheduling the Pipeline

Once the summarize-and-render steps are solid functions, wiring them into a scheduler is the easy part. A simple cron-triggered script works for most teams before you need a full orchestrator:

# run_ragas_report.py
import os
from ragas_pipeline import run_evaluation, summarize_run, render_markdown_report
from storage import load_previous_summary, save_summary
from notifiers import post_to_slack, send_email_report

def main():
    config = {
        "git_commit": os.environ.get("GIT_COMMIT", "unknown"),
        "pipeline_version": "v1.4.0",
        "eval_dataset_version": "golden_v3",
    }

    df, run_meta = run_evaluation(config)
    previous = load_previous_summary()
    summary = summarize_run(df, previous_summary=previous)

    report_md = render_markdown_report(summary, run_meta)
    save_summary(run_meta["run_id"], summary, run_meta)

    post_to_slack(channel="#rag-quality", markdown=report_md)

    if any(v is not None and v < -0.03 for v in summary["deltas"].values()):
        send_email_report(
            to=["eng-leads@company.com"],
            subject="Ragas regression detected",
            body=report_md,
        )

if __name__ == "__main__":
    main()

The regression-triggered email is deliberate: don't email stakeholders every day regardless of signal, or they'll filter the report into a folder they never open. Post routine reports to a low-friction channel like Slack, and reserve email (or a paged alert) for the moment a metric crosses a real threshold. This mirrors how mature teams treat CI notifications — noisy channels get ignored, signal-only channels get read.

For the actual scheduling mechanism, a GitHub Actions cron job, an Airflow DAG, or a plain crontab entry all work. What matters is that the trigger is deterministic and the config (dataset version, model version) is captured at run time, not assumed.

# .github/workflows/ragas-nightly.yml
name: Nightly Ragas Report
on:
  schedule:
    - cron: "0 6 * * *"
  workflow_dispatch: {}

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt
      - run: python run_ragas_report.py
        env:
          GIT_COMMIT: ${{ github.sha }}
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }}

Running this nightly against a fixed golden dataset gives you a clean signal: any score movement is attributable to a code or prompt change, not dataset drift, because the dataset is frozen and versioned separately.

It's worth adding a workflow_dispatch trigger, as shown above, even though the schedule is the primary entry point. Stakeholder reviews rarely happen exactly on your cron schedule — someone will ask "can we get fresh numbers before the 2pm meeting," and being able to manually kick the same pipeline from the GitHub Actions UI (or a Slack slash command wired to the same workflow) saves you from ever hand-running a notebook again. The moment you find yourself SSH-ing into a machine to regenerate a report by hand, the automation has failed its purpose, so it's worth treating "can any team member trigger a fresh report without touching code" as a real acceptance criterion for this pipeline, not a nice-to-have.

Making the Report Actually Land With Non-Technical Stakeholders

Getting the numbers into Slack is necessary but not sufficient. The format and cadence of delivery matters almost as much as the underlying data quality, because a report that's technically correct but poorly timed or poorly framed will still get ignored.

A few adjustments make a measurable difference in whether stakeholders actually engage with the report:

• Lead with the delta, not the absolute number. "Faithfulness: 0.81" tells a PM nothing without context. "Faithfulness held steady at 0.81, no change from last week" or "Faithfulness dropped from 0.87 to 0.81 after Tuesday's retriever swap" tells a story they can act on.
• Translate metric names once, up front, and stop re-explaining them every report. Pin a short glossary message at the top of the Slack channel — "faithfulness = does the answer stick to what the retrieved context actually says" — so the weekly report itself can stay terse.
• Match cadence to how the team makes decisions. A team shipping prompt changes daily needs a daily report. A team on a two-week sprint cadence mostly needs a report that lines up with sprint review, plus regression alerts in between. Reporting more often than decisions get made just creates noise stakeholders learn to skim past.
• Include one plain-English sentence a non-engineer could repeat in a meeting. Something like: "Answer quality is stable this week; one edge case around multi-turn questions needs follow-up." This is the sentence that actually gets forwarded up the chain, so it's worth writing deliberately rather than letting the bullet list speak for itself.

def one_line_summary(summary: dict) -> str:
    worst_metric = min(summary["deltas"], key=lambda m: summary["deltas"].get(m, 0) or 0)
    worst_delta = summary["deltas"].get(worst_metric)

    if worst_delta is not None and worst_delta < -0.03:
        return f"Quality regression detected in {worst_metric} (down {abs(worst_delta):.2f}). Investigate before next release."
    if all(v is not None and v >= -0.01 for v in summary["deltas"].values()):
        return "All metrics stable or improved since last run. No action needed."
    return "Minor fluctuations observed, within normal range. No action needed."

Generating this sentence programmatically and pinning it to the very top of the Slack message means even a stakeholder who reads nothing else still gets the one fact that matters.

Persisting History So You Get Trends, Not Snapshots

A single report answers "how are we doing today." A stakeholder review needs "how are we trending over the last month," which means every run's summary needs to land somewhere queryable. You don't need a data warehouse for this — a simple table is enough to start.

# storage.py
import sqlite3
import json

DB_PATH = "ragas_history.db"

def init_db():
    conn = sqlite3.connect(DB_PATH)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS run_summaries (
            run_id TEXT PRIMARY KEY,
            timestamp TEXT,
            git_commit TEXT,
            summary_json TEXT
        )
    """)
    conn.commit()
    conn.close()

def save_summary(run_id: str, summary: dict, run_meta: dict):
    conn = sqlite3.connect(DB_PATH)
    conn.execute(
        "INSERT INTO run_summaries VALUES (?, ?, ?, ?)",
        (run_id, run_meta["timestamp"], run_meta.get("git_commit"), json.dumps(summary)),
    )
    conn.commit()
    conn.close()

def load_previous_summary() -> dict | None:
    conn = sqlite3.connect(DB_PATH)
    row = conn.execute(
        "SELECT summary_json FROM run_summaries ORDER BY timestamp DESC LIMIT 1"
    ).fetchone()
    conn.close()
    return json.loads(row[0]) if row else None

With even a few weeks of history in this table, a weekly stakeholder review becomes a chart, not a paragraph: faithfulness over time, context precision over time, with commit markers where major pipeline changes landed. That's the artifact that actually changes how a leadership team behaves — they stop asking "is the RAG system good" as a yes/no question and start asking "why did precision dip in week three," which is a much more productive conversation.

If your team grows past a handful of dashboards, swapping SQLite for Postgres and plotting through something like Grafana or a simple Streamlit app is a natural next step — the schema above ports over directly, it's just a matter of scale.

Handling the LLM-as-Judge Cost and Reliability Problem

Ragas's default metrics rely on an LLM judge, which introduces two automation-specific problems: cost and non-determinism. If you're running this nightly against a growing golden set, judge-model API calls add up, and a judge model that gives slightly different scores on identical inputs will make your trend lines noisy in a way that looks like real regression.

A few practical mitigations:

• Pin the judge model explicitly (llm_judge_model in your run metadata) and only change it deliberately, logging the change as an annotation on your trend chart.
• Set temperature=0 on the judge LLM wherever the wrapper exposes it, to reduce run-to-run scoring noise.
• Cache judge responses keyed by (question, answer, context_hash) so re-running the same test case against an unchanged pipeline doesn't re-spend tokens.
• Run a smaller "smoke" subset (30-50 cases) nightly for fast signal, and the full golden set weekly for the stakeholder-facing report — this keeps daily cost low while still giving you a comprehensive weekly number.

import hashlib

def cache_key(question: str, answer: str, contexts: list[str]) -> str:
    context_hash = hashlib.sha256("".join(contexts).encode()).hexdigest()[:16]
    raw = f"{question}|{answer}|{context_hash}"
    return hashlib.sha256(raw.encode()).hexdigest()

This caching layer alone often cuts nightly evaluation cost by more than half once a codebase stabilizes, since most test cases don't change between runs unless the retriever or prompt actually changed.

Common Pitfalls When Automating This

A few mistakes show up repeatedly when teams build this pipeline for the first time.

• Comparing runs against a moving dataset. If someone edits the golden set between runs without bumping eval_dataset_version, your delta calculation silently becomes meaningless. Treat the dataset file like production code — PR review required.
• Reporting averages without sample size context. An average from 15 test cases and an average from 400 test cases are not the same kind of evidence. Always show sample_size next to the score.
• No owner for the "worst cases" list. If nobody is accountable for looking at the five worst-performing examples each week, that section of the report becomes decoration. Assign it in the same way you'd assign a flaky-test owner in CI.
• Alert fatigue from over-notifying. If every 1% wiggle triggers a Slack ping, the channel gets muted within two weeks. Threshold your alerts, not your reports.
• Treating the judge model as ground truth. Ragas scores are a proxy, not a verdict. Pair automated reports with periodic human spot-checks, especially before using a Ragas trend line to justify a go/no-go ship decision.

Bringing It Together

None of the pieces here are individually complicated — a summarization function, a Markdown renderer, a cron trigger, a SQLite table, a Slack webhook. The value is in wiring them together consistently so that "checking on RAG quality" stops being a manual notebook exercise and becomes something that shows up in a channel every morning whether or not anyone remembers to run it. That reliability is what actually builds stakeholder trust: not a single impressive score, but a report that shows up on schedule, flags regressions honestly, and lets a non-technical reader understand system health in under a minute.

Start small. Wire up the summary function and a Slack post first. Add the SQLite history table once you have more than a week of runs to compare. Add email escalation only once you've defined what a real regression threshold looks like for your product. Each layer is optional, but together they turn Ragas from a debugging tool you run by hand into a reporting system your whole team can rely on.

If you want a guided, hands-on walkthrough of building evaluation pipelines like this from scratch — including deeper dives into custom metrics, dataset curation, and judge-model reliability — check out the Ragas Tutorial course on teachyou.ai, where we build this exact reporting pipeline step by step alongside the core Ragas metric internals.